Interviewing a leading expert in performance engineering and open source tools

Recently for his Quality Sense Podcast, Federico Toledo interviewed Refael Botbol, the BlazeMeter testing domain expert at Broadcom, where he enables developers to achieve higher-quality applications by injecting testing throughout the software development lifecycle. He’s passionate about helping organizations democratize access to testing tools (performance, functional), allowing development teams to successfully achieve sprint deadlines. Refael has nearly 15 years of end-to-end experience, ranging from development and system engineering up to ensuring delivery of high quality applications.

Listen to their 40-minute conversation that’s been divided over two podcast episodes, or check out the transcript below!

What to Expect?

• Why perform shift-left and shift-right testing and also dare to run load testing in production
• Why giants like Netflix and Google choose open source tools and how you can make the transition easier
• Problems and myths of performance testing and the cloud

Most importantly, listen to find out which 1980’s SciFi novel Refael thinks predicted Jeff Bezos’ two pizza teams, how to know you are a software tester for sure, and how to REALLY become a trusted software engineer inside of BlazeMeter!

Listen Here:

Part 1

Part 2

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts

Episodes Transcript

Federico:

Hello Refael. Thank you so much for participating in this show. How are you today?

Refael:

Great Federico, thank you for having me. It’s super fun to do this with you. Now we can share what we are doing in the last couple of years together.

Federico:

Yeah, that’s true. And there’s something that I really miss during this lockdown situation because of COVID-19, which is visiting you in your office and having the opportunity to try one of the coffees that you prepare, the cappuccinos with this beautiful latte art that you’ve mastered.

Refael:

Yeah. In my company, BlazeMeter, we have a tradition that first, you’re some kind of a UPS delivery person because we have multiple sites. So, every time you visit a site, you need to bring something to Israel or to Russia, or somewhere in Europe. 

Then, you become a barista because you need to know how to make good coffee, otherwise you won’t have good code. And only then you become a developer, a performance engineer. That’s why we can trust you, if you know how to do great coffee.

Federico:

Well, I’m far away from that.

Refael:

Hey, you can join in our training. That’s fine! Maybe not today.

Federico:

I will! I will! Well, the main topic I would like to address today with you is related to how to optimize costs in software testing, particularly in performance testing. Because I know that you have a great background, having worked for a startup (BlazeMeter) which was acquired by a corporation (CA Technologies), which was acquired by a bigger corporation (Broadcom).

I understand that in your experience, you have been working in different contexts with different companies in different situations. So probably from that I guess you have a lot of things to share, which I believe is a topic very relevant for nowadays with this situation. So what do you think about that?

Refael:

Well. One thing I learned with BlazeMeter is that the industry always surprises me. When we started in BlazeMeter I thought only the big companies, the Fortune 100 need those high [inaudible] and I discovered that it’s actually the opposite. Or let’s say, you’d be surprised how a startup can challenge your product in a way that no one has challenged it before. And it’s fun. It’s actually very exciting to everyone to get that challenge of, you need to drive the network, the KPIs network. So how would you do that?

We coined continuous testing and shift-left / shift-right, but people didn’t really understand that by reusing the tests, that the assets of a performance test that you worked so hard to create, you can help your developers, QA, even operation guys do experiments and get faster results.

And I’m saying experiment because if you think about it, every time you write code, you’re doing an experiment, right? It’s like art. You’re trying something but, different to art, you can really easily change and do drafts and many drafts. You’re trying something and you want to get quick results to know if what you did makes sense, not does it perform, do you have issues. And we see more and more organizations and I’m happy to say that also startups are taking that from day one, which is great to see because it will bubble up.

And implementing tests, performance tests all the way, even in the integration environment, even on your laptop… not going to be the full blown… But the idea is that if you take those assets and you run them, then you are able to create first a baseline, second a validation that the business flow is still working. Because hey, it worked for me before my code change and after my code change. Or I upgraded my database and I want to make sure that I get the same kind of trend. And it’s good, something that’s like you check your pulse and say, it makes sense. And then the feedback is faster.

So I think that’s one of the things that really helps organizations push and be more effective because you find issues faster and we all know that if you deployed it and it’s already now in staging and that’s the first time you do a performance test, and you find a functional issue, you’re going to waste a lot of time on that. One, is the time the performance engineers will assume it’s not a functional issue and they will try to figure out what happened there.

Second, once they will figure out it’s a functional issue, then it will take time for them to convey all the knowledge to the developers. And then the developers will start working on that. If we shift left (or also shift right, we can talk about it later), the developers first get to own that. Second, they know that way before and they can do the change, you don’t even need to stress your performance engineers for that.

Federico:

Yeah. When you were talking, I remembered a talk you gave at a conference that I saw related to shift left performance testing. Which is I think one of the ways that we have in order to reduce costs or maybe to optimize them. I like the idea of optimizing costs more than reducing costs because I think we can get much more value from what we are already investing, just by making some slight tweaks to the way we’re doing things.

Like if we start testing earlier as you mentioned, I think we can get many more results and benefits from what we do. I really liked this idea and typically when people speak about shift left testing, they think mainly in automation or functional testing, but it also applies to performance testing, right?

Refael:

Yeah. In my opinion, it’s a shift of mind. You look at your developers not as people that write code. They own the application and it’s our job to empower them to be able to do that. Now, if I’m a developer, should I know all the bits and bytes of my database? Obviously not. But I should be able to connect to that and somebody needs to facilitate that for me. And if I have a requirement from the database, I can go to my database manager and kind of talk with them and kind of see what we can do with that.

Same goes for security, right? If I’m a developer and I want to write code, I want my code to be secured. I’ll do my best to know the best practices. I’ll make sure that my NPM packages are checked with something like Snake or make sure that I’m following best practices. I will become knowledgeable about SQL injection and we’ll implement something because I want my code to be secured. However, the organization will not expect me to go and do the penetration testing, right? It’s a different mindset of different skills.

So same goes for everything that goes with quality. The end goal is quality. Testing is one way to do that, and it’s not the only way. But it’s something that will… You talked about reducing costs versus getting more. I think it’s just the return of investment question, right?

Just like in soccer, if you have the best defending team without the attacking part of it, then you’re not going to win. If you have just attacking but you don’t have defense, you’re not going to win as well. So it’s kind of a balance that you need to get in order to get the most for your investment.

Federico:

Sure. That’s true. I really like the analogy with one of the most popular sports in Latin America.

Refael:

You guys have Suarez, so you know…

Federico:

Yeah, that’s true. Okay. Well there’s also some things that typically happen. Thinking more on the right side of the process, if we start shifting right, our mind. Something that I see typically is that when you find problems in production, you just put more hardware.

You increase the hardware. And that increases the costs, right? So probably there are little things that we can do there on the right side of the process in order to optimize also. What’s your view or your insights about that?

Refael:

It’s funny, your timing for that question is interesting. Two things. One is just before the whole COVID-19 started, I talked to my colleague and friend Feloney, Steve Feloney. We talked about the days when cloud just started and people said, “Oh, that’s it. You don’t need performance testing. What you need to do is just move to the cloud.” And he told me that he had a talk then saying, but you can’t scale forever, right? What will you do? And are you willing to pay all that additional cost?

So I think history said that performance testing is still here even though we’re in the cloud. And I see startups that actually are helping people to optimize their hardware. So, and what optimize is, I don’t know. It can be money, it can be throughput, it can be standardization. But they allow you to use your performance test in order to check a lot of your configuration and then suggest you the best optimal configuration based on the KPIs that you had.

And one of them is a startup that I did a webinar with called The Consortium. And it was like a perfect fit.

Because now, yeah you can scale, but you need to go and standardize things and you need to make the application be able to do that and how/which configuration to do based on the application. And I think we’re going to see more and more of that happening. And also with regards to the shift right, every customer that did shift left, the second they figured that out and saw the value, they are like, how can I run that in production? Because then you can create the baseline versus the production data. 

And it’s like synthetic monitoring but not for the monitoring and Ops guys, for the performance engineers, right?

So if you’re a performance engineer and you’re responsible for, let’s say a March Madness or Black Friday or a Fantasy Football, something like that. The first thing is going, okay, what is the scale? How do I know what are the test criteria? And usually you need to work with Ops and APM like New Relic and CA Digital Experience Insights and App Dynamics to kind of figure out the business flow and what are the KPIs for that. It’s a lot of work.

But they can just go now into the same tool that they use and look at the same business flow that they run as a performance and they’re running their integration and staging and pre production, and see it in real time and just grab those KPIs and say those are my KPIs, and run the performance test.

Federico:

Yeah. The closer you are to the user, the shorter the gap. The smaller the gap between the testing environment and the real use of the application, right? So-

Refael:

Yeah.

Federico:

Yeah. Decreasing this, working in this gap, reducing this gap, I think it’s their way of reducing the risk because you reduce the uncertainty or the questions you have about how it will work in production. But every time I talk about testing in production, people hesitate.

It’s like, “Oh, how can I do that? I cannot take resources from the servers that my users are using.” So how can companies do that? Or is there any particular thing to take into account in order to run effectively testing in production?

Refael:

Let’s see. I think it all starts with the decision of are we going to test in production or not? Because if you decide that it’s not important to you, you will find all the excuses, right? Some actually have good technical issues of regulation, right? So it’s not that they don’t want to try to figure out how they don’t get the private information stored somewhere in the cloud in [inaudible 00:15:31].

But apart from that and there are ways to deal with that, it’s just the most complicated case of that every company can test in production. 

I’m saying to them that if you don’t test in production, you’re guessing. 

You might have a really good guessing game and you might be the best guesser in the world, but you’re guessing. Why? Because even if your environment is 100%, if you didn’t test your actual production or what is about to be production, you could fail just because of a configuration issue. Or just because somebody forgot to switch a flag somewhere or the code was a bit different and those kinds of things.

We saw it with all of our customers. And the beautiful thing about BlazeMeter, it’s the product that caters to five people in a garage startup and to a Fortune 100 company that needs to go full-blown. So we see all the markets and all the needs there. And all the companies that we work with, most of them, let’s say, most of them are doing it in production. Because today you have technologies that allow you to do blue-green for example. So you can do that.

People are already embedding maintenance windows into their SLAs. So they are having those maintenance windows and there are a lot of ways that you can test in production without interfering with your current users or production environment, or at least minimize that. So big e-commerce, big banks, telecommunication companies, they all could’ve said, “Well, we need to be 100% or we will lose money, we will do that.”

And they all test in production. They don’t get charged by their customers on the SLA and those kinds of things. So, can you do that? Yes. But it all starts with the, hey, we need to do that because otherwise we’re guessing.

Federico:

Yeah, with a decision. Someone told me once that if you don’t test in production, your users will.

Refael:

That’s true. If you don’t test, your user will be your tester. Now the question, who do you want to fail?

Federico:

Yes, exactly. Who do you want to find the failures for you?

Refael:

But it’s true that for example, all of us know about canary testing, which allows us to get… And I think canary testing is great, but it’s not a replacement for a performance. There are a lot of things that we used to do only via performance, and now we can do with canary testing.

It also requires the organization to be mature. But I remember on one of our QBRs, we invited one of the top performance engineers in Netflix… one of the advantages of being in the Bay Area.

Federico:

Yes.

Refael:

Somebody in the crowd asked him, “So you guys don’t do performance testing because you do canary so well?” And he said no. He basically said, “No. There are some things that now we can do better with canary and mitigate the risk. But we do performance.” And there are some things you cannot do with canary.

So for example, if you have three servers for your microservice, you cannot really do canary testing. If you have 100 or 500 users, it’s really tough to do canary. To do canary, you need to have thousands and thousands of users. So you’ll be able to do something which is give you the impact that you want, but not impact your users. It’s too much.

Federico:

Understood. Okay. And one last question related to the topic. What about open source? Because I understand that there are beautiful open source tools for doing performance testing and there are some companies already using commercial tools and things, some licenses around that.

And they may not want to try the open source solutions, mainly because they see that there is a huge investment maybe in the first stage like migrating all of what they currently have to the new tool. So I don’t know if you have any experience you can share about any company trying to transition to open source.

Refael:

Yeah, this is a good topic to discuss. We see a lot of enterprises going open source first. I think it will not shock people that will hear this that the VP level or the C-level will say, “Is there an open source tool that can do that?” If the answer is yes, then, it’s “So is there somebody backing up that open source? Is it healthy, is it being maintained?” And if the answer is yes, then that will be the first thing that we’ll try to do.

And I think the reason organizations, not startups, I’m saying startup is the obvious part, but enterprise, financial health insurance, they do that because one, it empowers them as an organization. Second, it’s really easy to start with open source. So if an open source is successful, that means there is a community around it. If there is a community around it and you want to get your users to use that, one, you know that they will be like any other users that are trying to do that and the community will support them.

Second, if the community is big, then you know it’s something that is easy. So they relieve a lot of the concerns of a new product just by going with open source. Second thing is the ability to do it themselves if they want to. And for us in BlazeMeter, we see it all the time. We see people try and try doing JMeter, build it yourself. And at some point, they get to the level that they spent a whole Ops team just to maintain that.

And we just tell them, listen, it’s much more cost effective to do that and you get all those features and it’s like a done deal. And we are happy to see that because for the same ROI, for the same resources that they invested, they got a lot more of their investment. So it’s an easy choice to go open source.

But if you’re an organization, you also want to know that tomorrow, I want to switch, tomorrow I want to do something. The scripts, because they are based on open source are mine, and I can move between one vendor to another or even build it myself, if it comes to that.

Because on the end, you don’t want a product to product comparison. You’re talking about relationships with the customer. You’re talking about how you help customers achieve their goals.

The tools are just the “how” to do that. And I think when somebody invests over $1 million in your product, they’re not buying the product really. They’re buying what you promise them. They’re buying the achievement that they will get with the partnership between you and the vendor. And the product is just the “how” to do that, as long as the product fits the needs of course.

Federico:

Yeah, sure.

Refael:

And by the way, forgot to mention, one of our competitors, it’s been there for a long, long time and we saw the problem of the market to move from that competitor’s tool to open source. 

So with the help of you guys at Abstracta, we created the LoadRunner or (as we call it) the shift-left converter to allow people to ease the pain of moving from LoadRunner to JMeter. It’s not a silver bullet, right? It’s not going to do all the work.

But we saw a couple of engagements that took a year and a half or even more than and reduced it to a matter of a couple of months.

Federico:

So that makes it an easier decision. And also there is another thing that you mentioned that I think is really important. When you decide to move to another tool, you have to pay a lot of attention to the community behind it. If it’s a company or if there are many people contributing to the community.

I am impressed particularly with JMeter, which is a tool that I’ve been using for many years. And you have everything you need. There are so many online trainings, there are people answering questions in different forums and things like this. So you don’t miss anything from a commercial tool if you use it.

Refael:

Yeah. And it’s fun to see that. Like look, Netflix released their CD tool for canary testing to deal with open source. They actually built on top of that another open source tool to kind of do the analysis of those canary tests and released that as well.

We see SmartBear with Swagger; they basically donated that and now it’s an open API project and more and more, we can talk about the Linux distributions and then people are going with open source because it’s just a win-win for everyone.

Federico:

Yeah, that’s awesome. Okay. Just to try to round up this interview, this episode of the podcast, I have a couple of more personal questions I’d like to ask.

One is related to habits. Because I really believe that a good way to improve things in your life/in your career is related to the small things that you do every day. So I am curious about which habits do you have or if you have any habit that you recommend our listeners try?

Refael:

Oof. I don’t know if it’s an everyday habit, but I’m trying to challenge myself with new things. And at least something in a half a year or a quarter. We talked about, for example, coffee, right? So one of my objectives was to find my coffee machine with the budget that I can and then become self-sufficient and roast my own beans.

One project led to another and we talked about how coffee is so scientific, you need to have the [inaudible] and you need to roast it in just exactly the same place. And the more you look into that, the more you understand you need to lay them for a week and calibrate your grinder. And it’s all started because there was a coffee machine in the office, right? And I wanted to make something there.

So from my point of view, think of the things you do daily and just how you can do them better and just make it a project for you. It can be your coffee and it can be your guitar, or whatever you find. And then just put your objective. I found it useful for me and it also helps me do it to my teams that we do. So for example, implementing OKRs as a practice can be something that you can do just for the sake of learning it and see if it fits for you.

Federico:

With the coffee?

Refael:

Oh, well. OKRs, it’s not just for work. You can do it for your own sake. And it’s good because with OKRs, you’re not supposed to be 100% and you’re not supposed to fail all the time. And it’s a learning process which you learn most about is yourself. Like today, if you tell me to develop a JMeter script, I will know exactly what to do and I will tell you how much time it will take.

But if you tell me today, “Hey, Refael, how much time will it take you to run a marathon?” I don’t know, right? But just doing that and putting an objective and see how you handle that and how you learn. You learn a lot about yourself. Is it physical? Is it the psychological where you work? And all you need is just that small objective. Okay, we want to run a marathon so we will run 10 kilometers in the next month.

Okay, you failed. You learned about it, that you’re not able to do 10K in one month and due to such and such reasons, right? So you learn a lot about yourself and that’s what I’m trying to do. Every quarter then I’ll find something and see if I can achieve that. I think Mark Zuckerberg is doing it per year and he learned Mandarin or something like that at one point. So he’s more advanced.

Federico:

Yeah. And it’s related to what you mentioned at the beginning. It’s doing experiments, right?

Refael:

Yeah. Sometimes you think you’ll need a whole project of renovating your house in order to do those kinds of things. I always dreamt of doing that and I see it as a dream that keeps on going because you never find the time to do that or the money to do that and those kinds of things. But daily, small things that you can say… Just for example, listening to X amount of podcasts. Or find a way to exercise, or learn about something that was always there.

And today, I don’t know about if you’re a manager, probably you’re overwhelmed with work. But maybe engineers today, because we’re working from home, we got our 2-3 hours of commute back. So instead of the podcast, maybe we can go to Coursera, go to Abstracta Academy or BlazeMeter University and learn JMeter, learn Selenium, learn performance testing. Maybe you’ll find out that you’re a tester!

Federico:

Yeah.

Refael:

And the tester is not just for code. You can test a lot of things in your house. How to pick a lock is also a test that you can learn.

Federico:

Everything is related to quality and analyzing the quality, improving the quality of everything around you. This is the life of a tester. 

Refael:

Somebody told me, “How do you become a tester? And I said, “You don’t become a tester. It’s something in your DNA.” If you want to know if you would like testing or not. If you’re a gamer, just go to any MMORPG game… sorry, massive multiplayer online game… or 3D Shooter. Just like Counter-Strike.

Refael:

Because you explore stuff and see what happens… “If I throw a grenade into that cave, will I die”?… Only a tester would think about that.

Federico:

Experiments. Experiments again. Okay. One final question: books. Do you often read books? Do you have any recommendations?

Refael:

One book that really influenced me is Ender’s Game, I don’t know if, are you familiar with Ender’s Game? It’s kind of an old book. Almost as old as me.

So Ender’s Game talks about… I don’t want to ruin the book but he talks about, there was an invasion of Earth by an alien race and now they’re training kids to become the next… really intelligent kids that they located. They’re moving them to space and the whole world was united to create that program and they teach them how to become officers to command the future army that will deal with the invasion.

I think that’s Ender’s Game is the book, in my opinion, that predicted the two pizza team.

Really quickly, in that book, the main character is Andrew Wiggins and his nickname is Ender. He is implementing squads, instead of just controlling everything as the other commanders. And basically, he breaks it down into squads of three, which they know the plan but they’re fully autonomous and they know their role because there’s a gaming space that they need to play and get to the goal.

I read that book just before I left to officer school, that book really taught me that it’s much more efficient to get your, in this case it was soldiers, but build your teams in such a way that they are fully autonomous and they own what they are supposed to do and they have all the permissions to do whatever they need in order to get that goal, obviously within the law, with the rules of the game. 

I think if I take that, and that book was written in 1985, if you look at Jeff Bezos’ two pizza teams, I think it’s just an extension of that.

You basically break your teams into groups that are not too big because then what’s the point, but not too small. They can do the work or the job all by themselves and they look at the, let’s say the CEO or the manager as the person that basically tells them, where they need to be, what is expected and what is the end game. 

So I love that book, and by the way, it’s a full series of books and there’s a game that develops into going into space and traveling between planets, etc. There is a parallel book series to that called Ender’s Shadow, which starts from almost the same place of Ender’s Game, but from a different character. In Ender’s Game, it’s from Andrew Wiggins’ point of view and then one of his squad, called Bean. Then Ender’s Shadow is from Bean’s point of view. Also great. I highly recommend those books, but be careful, you become addicted to them. Maybe start reading them on the weekend, so that you’ve got at least two days in which you won’t need to get up early for work!

Federico:

Okay. I will take this into consideration! At the beginning, I thought that you were mentioning a SciFi book; nothing to do with what we do at work… you know, but at the end, you connected it nicely.

Refael:

The writer I think just, I don’t know if by accident or not but he learned it from somewhere else, that’s how I looked at it, and when the two pizza phrase came up in the industry, I automatically thought, it’s Ender’s game but with eight people, not four.

Federico:

Cool. Excellent. Refael, I had a great time talking with you. Thank you so much for your time. The last thing I want to ask you is if you have anything you want to suggest to our listeners to access or try. Any invitation you would like to make?

Refael:

So I urge all of you, if you ever wanted to kind of know how to do testing with JMeter or become a performance tester, to visit BlazeMeter University. It’s free. 

There are learning paths to become a performance engineer which will guide you through how to be a JMeter pro, and then what it means to be performance engineers, how to conduct testing best practices.

It’s all based on JMeter, Selenium and the BlazeMeter free account. So with all those three, you can just do that. And if you need more into that, you have Abstracta Academy. I think those two together are completing each other.

Other than that, just, you know, stay safe! Stay healthy! In these crazy times, we learn so much about ourselves. 

So that’s the state of mind, and we’re all doing our best and learning to do that.

Federico:

Very challenging times, but I think we can do it. Thank you. Thank you so much again. Refael and talk to you soon!

Refael:

Thank you, Federico. Bye.

Did you enjoy this episode of Quality Sense? Check out other Abstracta podcast blog posts or check out the show on Soundcloud!

Recommended for You

Quality Sense Podcast: Rob Sabourin – Testing Under Pressure (Part One)
Shift-Left Testing in the Enterprise and the Case for Open Source