What do restaurants and performance testing have in common?

We’ll spoil the punchline…. It’s servers that get stressed!

In this episode of our software testing podcast, Federico interviews Leandro Melendez, whom he refers to as a “Latin American brother” also known as “Señor Performo.” Originally from Mexico, he’s an avid performance test manager and consultant with almost 20 years of experience in the software industry. He’s also a podcast host on performance, and a testing conference speaker. On top of all that, he generates a lot of content on performance in both English and Spanish. For example, he hosts the PerfBytes podcast in Spanish.

What to Expect?

• Performance testing concepts explained through funny analogies like flipping burgers
• Misconceptions around performance testing concepts and the role of automation
• Resources for those who want to understand more about software testing and performance
• Leandro’s morning routine and tips for staying productive, especially while working from home

Listen Here:

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts

Episode Transcript:

(Lightly edited for clarity.)

Federico:

Hello, Leandro. Hello, Señor Performo! This is so exciting for me. I’m so happy to have you here in the show. Welcome. How are you today?

Leandro:

Hello, amigo, Federico. Thank you very much for having me here. Super excited. I’m doing well, staying safe, healthy, and all those types of things. And really, really happy to finally have the opportunity to share with you on your awesome podcast. I am a big follower of what you have been doing and I’m really happy about it and very excited to be here.

Federico:

Thank you for your words, first of all. I think this is something that we have in common. I also follow your work, all of your publications and what you do, your podcast, your blogs and everything. I think what we have in common is that we are trying, correct me if I’m wrong, but we are trying to build a bridge between the English speaking world and the Spanish speaking world, trying to connect people and share knowledge from one world to the other. Is that what you are trying to do as well?

Leandro:

Well, to be honest, my initial objective for becoming “Señor Performo” and starting all these things that I’ve been trying to do was to help the world to fix many problems and confusions that I noticed within the industry at multiple clients, customers, organizations that I have been at. I often had to explain things, simple, basic concepts to management, even some seasoned performance testers and testers in general.

I was like “I don’t think that you understand what you’re saying or that’s the right approach or… ” and very simple, basic concepts so… And I was often explaining to people those basic concepts. I discovered that I had more or less an ability to do silly examples, basic examples, so that people would understand them, but I kept repeating and repeating and repeating them. So I was like, “I need to write this down in a way that more people can reach it and try to help the world to be ‘in peace’ and understand better testing and performance.”

And thanks to Mark Tomlinson (because I started that in English, most of the market that I could reach was in English), but Mark told me, “Hey, would you like to do the same, but in Spanish and jump that language barrier?” And I was like, “Oh, wow. Yeah. Awesome.”

It became like my second crusade or my second quest to try to jump that. And while doing that, I noticed that our Spanish speaking brothers and sisters were having trouble. Especially because most of the content was in English. And sadly, some of them have no access to the English language as you see some of us do. So I was like, “Well, I’m going to try to help crossing those bridges as well and sharing some of the experience,” and sometimes even plainly translating some of the concepts that I find, not only mine, but from someone else, bring them to Spanish and help people.

Federico:

Yeah, really cool. I think this is really, really valuable. So you told me that you’re writing a book right now.

Leandro:

Yeah. Well, writing right now, no. Yes and no! I have been writing it for a few years now and I am about to be done with it. I already have readable drafts that I’ll be happy to share with you and that are in the works, I’m polishing a little bit my Tarzan-like English. But as soon as that is done, I’m planning on trying to find some ways to publish the book and to help more people. 

The book is about load tests, not performance testing in general, but specifically load tests and trying to explain the little details of a load testing project. Waterfall-ish project, I know that’s old fashioned, but you always have to do load tests, big time load tests that you cannot follow too much the agile principles with those. So it’s a little bit retro, but there are concepts that are important for people to understand.

And as I mentioned, I tried to explain them with good examples, silly jokes, my mother throwing a “chancla” (Spanish for flip-flop) at me or eating some tacos at a restaurant, training dogs in the Cesar Milan style, the Dog Whisperer… So the book is filled up with examples. And after each example, I explain it in the technical—I call it the “official mumbo jumbo.” What you should call it, this is a test case. This is a formal performance test plan. This is the document, the groups, etc all the official lingo so that people that read it don’t only learn tacos and “chanclas” and learn a little bit more about load testing projects.

Federico:

Sounds interesting. I want to read it. So, to start talking about something related to what you are mentioning right now, what about explaining or visiting the different basic concepts around performance testing and load testing? I think it would be great to address those topics in this episode.

Leandro:

Yeah, of course. I am so eager to explain that all the time.

Federico:

Great.

Leandro:

Because you’re just touching a hotspot for me right now because that has been like my crusade lately to try to explain and it drives me crazy, I have to admit, that people often confuse performance testing with load testing and performance, they are like subfamilies. 

And on top of that, I am trying to create the topic or the term: “performance assurance” because we have quality assurance, which includes testing, but also tuning. In performance, the terms are all messed up and everybody confuses them.

Also, people tend to misunderstand performance testing with just doing some automations and slamming a system or doing an automation of a process that probably you don’t need to. The main goal for an automation for load testing is load. You are not going to automate a process that happens once a month, once a week, even once a day, it’s like why are you automating it? Just manually execute it, get the reading.

But, people would say, “Hey, but without that automation, how am I going to know the response time?” Well, there are multiple ways that we can do that nowadays. Probably 10, 15 years ago, that was the only way. But right now, we have APMs, we have nCode. We have telemetry. We have lots of things that you can do.

But back to the question, those terms, load testing is, again, using most of the time automation to generate a lot of activity to test the upper limits of the application load. That’s it. Some load tests (and some people will cry a little here) can be done manually. By people manually doing them if the load test, let’s say, it’s just you have a system with 40 users on it. Well, why not leverage UAT and other processes to do your load tests and just monitor the application? Another one is that you can use these modern release practices, send it to production in a blue-green environment, make your users do your load tests and you don’t need automations.

So performance testing and load testing is not only scripting and slamming. There are multiple ways. And again, performance testing, you can be testing the performance of a system with no activity to see how is the RAM, how is the CPU when no one is in the system and how is the “floor” consumption? That’s an important test, performance test. 

You can see when everybody leaves, how fast is the container decommissioned? How fast are environments turned off? So there are multiple performance test activities that you can do without actual load. Or as a single user, let’s say I’m the only one clicking this and I need to know if this process is fast.

I know that you have had some performance testing experience. How many times have you been asked to automate a process that is visibly slow already?

Federico:

Yeah, many times, many, many times.

Leandro:

Many times. And many of those, I’m like, “Why are you asking me to automate this that you already know has a performance problem and probably doesn’t even execute too often, happens twice a day?” 

Fix it, monitor it and tune it. You did your performance test manually. You don’t need an automation. You don’t need to do some of those old-time practices. I don’t know where the problem started. If it was two vendors… because two vendors are very happy that you tried to automate everything. And some service companies because I’m from a service company and I’m very happy when customers ask us to automate a lot of things because it gives me some more work!

But in all honesty, it’s like you don’t really need this. Your resources would be better allocated if you would ask someone else to just manually execute it, get the reading with an APM or some other solution and be done with it. 

Federico:

Yeah, maybe it’s not the best moment to do that. Maybe you can start improving the system without any automation and once you improve the system with only one person accessing it, then you can go to the next stage where you are going to simulate multiple concurrent users. Right?

Leandro:

I’m going to reutilize a little bit the example that I did with our friend, Henry from the Neotys PAC that we were at. 

You shouldn’t test those pieces separately, not in a race, just maybe on a parking lot or something like that at a low scale. And not even with automation, you don’t need the actual “track” and an actual “driver” and an actual automation for that. Just grab yourself the car, get it around the block and make sure that each of the pieces work together well. And even some of those automations, you don’t need to wait for the testers to do that. Teach the developer to put a start timer, end timer inside of the code. And there, even before they check in on the backend some of the processes, you can know that there’s a problem and avoid them to check it in.

So there are multiple things that I believe cause a wide confusion in terms of performance testing, load testing, performance assurance, where we hopefully, (all the industry organizations and people) will start to do things differently, rather than just trying to kill a fly with a cannonball.

Federico:

There’s another misconception I think which is when you ask someone what is performance about, they typically say velocity, how fast a system can be, but it’s not only related to response times. It’s also related to the resources needed to provide this fast response, right?

If you have a very fast response, but it takes 100% of your CPU or your memory or whatever, you don’t have good performance.

Leandro:

You could be in trouble, but it depends as well.

I like to define these three pieces of performance testing: 

• First, the response times, not only from the applications, you can measure response times everywhere as I said, a Docker environment, how fast does it come up? How quickly does it recover? That’s part of the timing. So one part is yes, time, how fast, everywhere, not only on a postback of a website.
• Second is to measure the impact of the activity in your system. As you say, how the CPU is impacted, how the RAM is consumed, how is the network, the four horsemen of performance: CPU, RAM, network and hard drive. How those are and even many others. It could be even what is the temperature of your computer? On some cell phones, it’s important to take that into account. I mean there are multiple things that you need to measure what happens.

• And the third element is a current circumstance that you want to test, if there are a thousand users concurrently, if there are no users, if there’s only a connection between two servers, what is the performance that you’re going to measure? 

So, yeah, it’s important as well to be aware and measure how the pieces of your solution are doing during a given circumstance. One key advice that I will try to give to testers: not all the circumstances at the same time, test one at a time. Otherwise, you are looking for a needle in multiple haystacks.

Federico:

Yeah. It’s like considering the variables one at a time. Yeah.

Leandro:

Yeah, yeah, yeah.

Federico:

Another topic where also, I typically see some confusion, is related to scripting and automation because when you mention automation to someone in the software industry, they automatically think about Selenium or this type of automation where you automate the actions of a user in the graphic user interface. 

But the automation we prepare for performance testing is totally different. Can you explain a little bit about those differences?

Leandro:

Well, following your example with automation, because the word, automation, is that you create a process to execute automatically that you don’t have to go and actually click on it. But as you mentioned, for many, an automation is a LoadRunner script, a JMeter script, a Selenium script and that is the automation. 

You can do an automation. I don’t know if you ever watched the Simpsons episode where Homer had the little bird that clicked the yes button.

Federico:

Yeah.

Leandro:

That is an automation. That was an automated process that was clicking yes. You don’t need a tool. And this, a little bit, like you say-

Federico:

Well, there is a tool involved in this example, right?

Leandro:

Not a software tool.

Federico:

No. Yeah, correct.

Leandro:

And you can create automations that way in multiple ways, but what I was trying to convey is that the way that you do them is not only through a tool, a LoadRunner or a JMeter, a Selenium, etcetera. There are multiple ones. And these automations, as you well said, when you talk about a document, everybody thinks right away about Microsoft Word, but there are G Docs. There’s Open Office. There are multiple ways, even our good old friend, the Notepad.

Federico:

Cool.

Leandro:

Plain and simple, and you can create a document on it. 

So in performance testing, I think one of the biggest confusions is between the automations for load generation and for, what I call, “pinging.”

I don’t know if there’s an official term, but it’s related to synthetic monitoring because there’s organic where you have someone in your system using it. 

You can monitor that activity, that performance. But let’s say that your site is down or for some reason, it’s extremely unpopular. No one logs in, you can not have that organic monitoring. So you create some scripts that here and there execute some actions so that you can measure it, even to measure if the system is online, alive. 

Because if a real user hits it and says that it’s down, they’ll leave and you won’t know. If you don’t have a mechanism for that, you won’t  know that it’s down. Most of the customers won’t call you and say, “Hey, I think your site is down.” We just curse at them and leave and open up Facebook and go do our daily business, right?

But if you have something that is trying to log in every 10 minutes, you will know within 10 minutes of the system going down, and this is part of automation in performance. You can receive a ping back, seeing how the activity is. And again, this is not related to load at all.

Now another confusion that I see with performance related automations is around the load testing principles, specifically between the browser and protocol.

That’s another one where many do not understand why it’s better to have it at a protocol level and not bring up a full browser window where other times, you do need the browser window. There’s no other way to automate. The protocol might be too encrypted, impossible.

So I give this example on those differences: You have this golden arches fast food restaurant where you have someone in the cashier, at the point of sale. The point of sale is your browser window. And when the cashier takes your order, he or she clicks something on the window, and it’s sent through a wire to a printer in the kitchen. That wire is the protocol and the printer and the kitchen and the server, well, is the server, the cook to call it in a way.

When we are automating at a protocol level, we are automating the messages that go to the printer. We don’t care about the cashier. We know that when the cashier clicks something, a message is sent, “Cook me a quarter pounder” and that’s a message that is received at the server. We can time how fast the quarter pounder appears in our front end to call it in a way, but we don’t care about the cashier.

If we do the load simulation there, it’s very easy to send multiple messages through the wire. It’s not expensive. You don’t need much. But if for some reason, if that wire is covered, protected, you have no way to insert the messages there, you need multiple cashiers, which are expensive, slow. You need the people to drive them or put some robots that will actually click on that front end. So those are the disadvantages. They are slow, heavy, bulky, costly, but that’s the difference. You have to render the whole protocol, the whole browser, your whole POS, your whole cashier, or you can just send the messages. Both of them are stressing the server or the cook in the kitchen. So more or less, those are the differences.

And the last one that I would go to is the backend testing, where instead of depending on the printer, you just stand in the kitchen and tell the cook, “Hey, five hamburgers right now. Let’s see how you do!” 

That I would say is API or service testing or even class or code performance testing where you tell him, “I won’t hire you yet. You’re a cook. I’m going to order you a quarter pounder and I’ll start my watch to see how fast you can cook it.” Even before you ask the chef to cook 10 per hour or something like that. So more or less, those are the examples that I use to explain those differences.

Federico:

Now I am starting to see the connections you make with performance testing and cooking and tacos and things like that!

Leandro:

The book is filled with those types of examples. Yeah, yeah. I can’t deny it!

Federico:

I love it.

Leandro:

And they help even for management to understand these types of differences, where for management, it’s just like:

“Yeah, use the tool I bought for you and do the testing thing.” 

Well, “Which testing thing?” 

“The performance thing.” 

“Which performance thing?” 

“Load testing, tuning, monitoring, what do you want me to do???” 

It’s a huge universe. So, those are more or less the examples that I generally go to. Silly, simple, but I think everyone more or less understands them, I hope.

Federico:

Yeah. I used to say that doing the simulation at a protocol level is more efficient and actually cheaper, let’s say, because you don’t need that much infrastructure in order to have all the browsers or all the things that you need in order to simulate the cashiers clicking the buttons, right? But it happened to us in some projects where the protocols were encrypted, it was very complicated. And for example, trying to imagine, now we are in a Zoom call… Trying to simulate multiple access with Zoom users and working in these “wires,” in these protocols, would be very complicated, very complex to do the reverse engineering and automate that. So in that type of case, it’s better to have the cashiers.

Leandro:

Yeah. Yeah. I mean that’s what I said. Sometimes the wire is totally isolated, underground. You have no ways. It’s encrypted. 

And even as I will challenge a little bit your Zoom example, Zoom works based on services. There are APIs for everything. On the backend, you can automate almost everything, but when it’s as complex as pretty much “The Matrix” code falling, and you have no idea what those green Chinese letters are saying… Oh, I’m sorry. I don’t know if they are Chinese. But that’s my point. I don’t know what they are. I cannot decode them, and it would be foolish to even try, and that’s why the browser protocols exist.

The problem is when this is called “man with a hammer” cognitive bias, when you learn to hammer well, everything will look like a nail. So many organizations find out that protocol scripting is easier and go crazy about it and run with a hammer, clicking nails everywhere. But there are moments where it is bulkier, more expensive, more difficult than some of the things that would be cheaper, faster, more efficient to be done with protocol. 

They are wasting a lot of resources involving the browser unnecessarily. So it would be like when would you use a screwdriver or a batter mixer? It depends. Both of them do more or less the same thing, but it depends on what you are needing and what’s the medium, the environment, the circumstances.

Federico:

The goal.

Leandro:

And that’s another one that I’m trying to help the world with those misunderstandings that you should be careful with. At an organization that I was trying to teach someone about performance testing, I drove the guy crazy because my responses were always: “It depends.” 

He wanted a formula. And I was like, “Man, performance is very difficult to go on with formulaic processes.” It always depends on what you’re trying to do. What is the goal? What is your concern? What is the risk that you’re trying to mitigate? It depends.

Federico:

Yeah, totally agree. Another question, what’s the most entertaining part for you in a performance test?

Leandro:

Oh, wow. I have to admit that it’s a part that many hate or run away from, and I tend to call it, to me, it’s like Stockholm syndrome, it’s scripting and correlating. 

I enjoy it a lot. I enjoy cracking those protocols and finding out what’s changing. Eventually, it became a very natural process that I just follow up, do my comparisons, try to find what are the types of things that are correlated, try to figure out which ones, why it’s not catching.

 But, it becomes a great pleasure to me when it’s like, “Oh, my God, it’s working and it never ever breaks.” I like to call them bulletproof scripts. Because even some organizations are like, “Yeah, my scripts fail at a 10%, 15% rate.” And I am scandalized when I see that. I’m like, “No, no, no. Zero failures. If they fail, it’s because the environment is starting to have problems.”

So I think scripting is probably what entertains me the most. The second thing would be looking at APM graphs to see how something impacts everywhere and dig through the problems in the APM tools. That’s very entertaining as well.

Federico:

Trying to find where the system broke, right?

Leandro:

Yeah, yeah, yeah. Where is that semi-colon that is causing all the problems inside of the code or the database or… Oh, my God. There’s a full table scan. That’s the whole problem. Oh, it’s fixed. So, that’s very entertaining as well.

Federico:

Excellent. It’s nice to see how other people also enjoy similar things. I really, really enjoy running performance tests and trying to understand complex things and find a small piece that is causing all the problems. 

In a different vein of questions, do you have any books to recommend?

Leandro:

In terms of testing or general?

Federico:

Whatever you want.

Leandro:

Well, in terms of testing, and this is not because I’m here. I generally—sadly, think you should work on having it in English as well. Your book, Federico, is pretty good. You have a wide coverage in general testing terms, not only performance. And I think given the small space that you had in the book, you cover pretty much everything, not too deeply because you do a quick general review, but I enjoyed it a lot.

In terms of right now, it’s very trendy, the agile testing movement. I like Janet Gregory and Lisa Crispin’s books a lot. The latest one that they published is really short, very introductory, and I would recommend starting with that. Not the first ones because the first ones are thick bibles of knowledge, very, very resourceful. But the newest one, I think it’s 150, 200 pages, quick read and very introductory. I love it.

And in the world of performance testing, I would say the Every Computer Performance Book from Bob Westcott, if I remember well. It’s very good. He also has a way to explain topics in simple manner, simple terms. I think he wrote the book 10, probably 20 years ago, but it’s simple to understand. He goes through similar examples, silly pictures, images, and it helped me to understand very well. He doesn’t go too deep or too technical and it’s for everyone who wants to get a better understanding.

Federico:

Another question I have for you is related to habits. Do you have any habits to suggest because I really believe that a way to improve your productivity or whatever you want to achieve, it’s very important to pay attention to those little details in your daily routine that you can improve. So what do you think it works for your routines for your habits?

Leandro:

Well, I think you just walked into an excess in terms of habits. I’m a very… I consciously try to hammer some of them into my life. 

I try to wake up as early as possible. I used to do that at 5:00 AM. Now, living together with my girlfriend, she doesn’t like that hour and I had to move it a little bit later. But the first thing that I do is get some coffee, start to read. I try to read at least 30 minutes every day, no excuse. And I’m keeping up a decent 20 to 50 books a year, try to keep them flowing.

After that, a quick workout routine where you also have to keep the “gut box” healthy. And after that, I try to do some learning outside of the books, at least. At our company, they gave us Udemy corporate access. So I try to do at least 10 minutes of something, even if it’s just learning about audio editing and engineering, testing, containers, Azure, whatever. I try to hit at least 10 minutes every day of that. It’s awesome that you can do that on your cell phone laying out on the couch and just be slacking and watching Udemy.

And after that, I do 30 minutes being creative with something. I record a podcast. I try to produce not just to be… as I love consuming stuff from the world, just to give something back. And I think those routines are very important. Especially in the lockdowns, if you don’t fall… When I started as a consultant that could be at home, I would wake up at 11 and you start to fall into depression and things like that.

So another thing that I do, and I recommend highly is shower early as if you were going to the office. Dress up. There are several psychological studies that you should dress for the work that you want to have and to do. 

If you’re in your pajamas, you are not as productive as if you have a shirt and a blazer or something, and you’re working. Even at home, it makes a huge difference. You feel yourself better, more productive. You flow better, call it in a way. So, that’s another one, be dressed up early. 

Even if you have to take a nap midday or something, you’re at home! Take a nap after lunch. I recommend that highly as well. I always take a 20- to 30-minute nap after lunch. It keeps you healthy and wakes you up a lot in the afternoon.

Federico:

Oh, man. That was a lot of tips! Thank you so much for sharing them. And as a last question, do you have anything you would like the listeners to try to subscribe to your podcast or YouTube channel or… ?

Leandro:

Well, yeah. Thank you. I’m going to do a shameless plug here to my YouTube channel. I started two YouTube channels not long ago. I’ve had the Señor Performo in Spanish channel for I think one or two years now, but I started a series called “Nosotros Los Performers” that’s in Spanish. But I decided, hey, I can produce this as well in English so… And it has like a theme of a Mexican telenovela. 

So I created the English channel where I’m going to try to put only English content, and there’s the Spanish channel, Señor Performo, just like that, it’s a Spanish one, Señor Performo ENG is the English channel where I’m going to start… Well, I already started the show, “Us, the Performers,” which is like the Mexican telenovela thing that I mentioned. 

I hope next Monday, we’ll be pushing the second episode and hopefully, I will keep the flow, publishing episodes in English and Spanish on both channels. That’s one. For the ones that speak Spanish, PerfBytes Español, we are producing and soon to have our friend, you, Federico, on it speaking in Spanish as well about testing.

Federico:

For sure!

Leandro:

And I think for now, that’s it. I haven’t been able to write that much lately, but there’s the Señor Performo blog. And here and there, webinars, presentations. 

Everything is posted on the social networks, Facebook, LinkedIn, YouTube, Instagram, Twitter. I think that’s pretty much everything. 

And any questions, comments, or just to say hi, [email protected] or [email protected], very welcome!

Federico:

Man, it was a pleasure. Un placer, realmente.

Leandro:

Thank you.

Federico:

Thank you so much and keep safe, healthy-

Leandro:

Thank you, amigo. You, too. And thank you very much for having me.

Federico:

Okay. Bye-bye.

Did you enjoy this episode of Quality Sense? Explore similar episodes here!

Recommended for You

How to Make a Performance Test Plan
Quality Sense Podcast: Paul-Henri Pillet – Why We Made Gatling