What it’s like to build a company around developer-driven performance testing

In this episode of the Quality Sense podcast, our COO, Federico Toledo, interviews Paul-Henri Pillet, a Frenchman and the CEO of one of our favorite open source load testing tools, Gatling, with 5,000,000 downloads in +100 countries to date.

What’s the Episode About?

Paul-Henri and Fede dive into the story behind Gatling: why it was created and what need it has filled for development teams and why he, someone without a technical background, was drawn to the project.

Paul-Henri also shares some of the most valuable advice he’s received as a CEO and, as in every episode, a book that has left a big impact on him that he recommends to others.

Listen Here

• Listen on Soundcloud
• Listen on Spotify

Episode Transcription

(There has been some slight editing for clarity.)

Federico:

Thank you so much, Paul-Henri, for accepting this invitation. How are you today?

Paul-Henri:

I’m fine, thank you very much, and thank you for the invitation also.

Federico:

Sure. How are you? Well, how are you, your team and how is the business doing in this lockdown situation because of the coronavirus?

Paul-Henri:

So everyone in the team is doing great. It’s a bit complicated to do business right now but we still have companies that contact us because they have huge performance issues. So I wouldn’t say it’s business as usual. It’s hard for everyone, but we manage to continue our activity and also we manage to work with everyone remotely. So that’s actually a good test for a company.

Federico:

Yeah. It’s part of the new normal, right, just stay at home and stay safe and take care of ourselves and the Earth.

Paul-Henri:

Exactly.

Federico:

What do you think is the biggest challenge for companies nowadays related to performance testing facing this new situation [the coronavirus]? Because you mentioned that maybe there are some companies looking for how to do performance tests or maybe needing some tooling for that. Do you have any suggestions or any insights related to the new needs related to performance testing?

Paul-Henri:

Basically, you’ve got two ways to do load tests. The first one is just to make a test on your existing application and to try to detect where your problems are, if you have any bottlenecks. That’s always the first steps you make when you start with a load test. It’s very useful. You should definitely do it, especially if you had recent performance issues with your application. But there is also another way of doing load tests for projects which is to do load tests on a regular basis and at the very beginning of your project.

What’s also very important with load testing is to try to detect as soon as possible all the problems that you can have in the future.

And it requires a lot of, how would you say, cooperation within your organization because you have to know what is your top priority, what are also your forecasts, if you plan to have [inaudible 00:03:02] zero risk, if you plan to have that kind of growth so, for instance, if you plan to have 10 times zero risk within one year.

And it’s very important for the development team to start to have these kinds of strategies, to know what kind of SLA they should have for the application, what kind of performance they should have.

That’s also what load testing is made for. So it’s not only just validating something, but it’s like a task that you integrate in your development life cycle.

Federico:

Totally.

Paul-Henri:

And that gives you guidelines for the development. So it’s really important to do both actually.

Federico:

I guess that with this current situation, there are specific companies or organizations paying more attention because they are having more traffic than usual-

Paul-Henri:

Yeah.

Federico:

I’ve seen many companies struggling with having to run performance tests from one day to the other because they didn’t plan this. It’s very difficult to forecast this type of situation. Right?

Paul-Henri:

Yeah, exactly.

Federico:

So how can you do a very quick performance test? Is there something to take into account to try to be ready or to try to get some results on optimizations of your infrastructure and your system as soon as possible?

Paul-Henri:

Sure. So the first thing you need to do is to find quick wins. In order to find quick wins, you need to find what are the things that have the most problems in your application, if it’s the infrastructure, if it’s your application, the architecture of your application, and load testing makes it possible to easily detect what is the biggest bottleneck in your application.

But load testing is not magic so you won’t solve everything with your first day you’re making, so there is also a long-term approach that you need to do also right now. Even if it’s a time of crisis and you have a lot of things to do, but if you need to develop new features very quickly, if you need to rethink your application architecture, you should start to do load testing right now when you start the development.

“It’s always hard to convince companies and development teams to say, “Okay, you’ve got a short-term approach for load testing and you’ve got also a long-term approach and you should do BOTH, even if we’re in a time of crisis.“

PAUL-HENRI PILLET

Federico:

So basically you have to be always ready for doing these types of tests or knowing about the performance of your system or how to improve it. It’s a continuous job, right?

Paul-Henri:

Yeah, and actually we had this situation quite often even before the crisis. For instance, during Black Friday, we had a lot of companies contacting us and saying, “Okay, it’s Black Friday in one month and we fear that we may have some problems with the performances.” The first thing we say is if you never tested your applications, you’re probably going to have problems. But it doesn’t mean it’s too late to do performance tests. You should do performance tests right now because when you’re going to deal with that kind of issue during Black Friday, it will be very useful to have that kind of insight. You will know how to react, you will know what are the components of your application that have most of the performance issues. So you should be able to deal more easily with that kind of situation.

Of course, if you need to do some refactoring performance tests, always a good ally to do that and to make the right technological choices. So yeah, it’s a bit too late but doesn’t mean you shouldn’t do it. You should start to do it and it would be also a good start for the future is to start to implement that kind of strategy in your development.

Federico:

Cool. Thank you. There is something else I’ve been thinking a lot lately also related to this global pandemic. This is the most talked about topic nowadays, but there are many companies that are struggling with the situation and analyzing how to reduce costs or trying to optimize the company in order to guarantee the business continuity. Do you have any insights, ideas, to share about how we can reduce the costs of software testing or maybe specifically related to performance? Anything you want to mention about that?

Paul-Henri:

Yeah, sure. So it’s a question we get a lot, what is the return on investment for load testing. The most obvious thing for load testing is when your application is crashing, what is the cost if that crashes? Especially in these times when it’s really valuable for companies to have that kind of selling channel so for any kind of business.

So that’s the first thing you should take into consideration, what is the financial impact of the crash.

Secondly, also when you are developing … This is a situation we often see. At the end of a project, people start to realize that there are performance issues and to solve the performance issue at the very end of a project can take some time and sometimes you have to start all over again because there are some highly critical issues. So this is why we also tell people to start early because it would be way easier and quicker to solve these kinds of problems if you detect them early.

Something also we struggle with because, as you can see, if you want to have load tests in a development team, if you want to have a load testing or strategy, you have to have some kind of political will and we need to talk with the top management and to convince them that there is an obvious return on investment regarding load testing at the beginning phase. At the end of your project it will be way more quicker and also you will avoid situations like crashing that you didn’t plan. That’s something we unfortunately saw too many times in these past few days. And public administrations, organizations and companies that simply weren’t ready to have that many people connecting at the same time on the application.

So for them, the losses are quite obvious. They’re losing a lot of users, probably losing a lot of opportunities-

Federico:

Yes.

Paul-Henri:

… due to the crisis. This is the kind of stuff we need to talk with everyone within [inaudible 00:11:20].

Federico:

I think there is a problem with performance testing that also happens with security or other quality factors that it’s really difficult to convince someone to pay attention to that until the moment you have problems related with this.

Paul-Henri:

Yes, exactly.

Federico:

You suffer financially… You feel an impact on your income or something like this. After that, you learn that it’s important, but it’s really hard to convince in the earliest stages for someone that hasn’t suffered before, right?

Paul-Henri:

Yes, and that’s the paradox also because in the development world I would say everyone has already experienced that kind of situation, but if there isn’t any collective will to deal with that, to tackle that issue, you can’t do anything because you would start to spend some days trying to do some tests and there are other priorities that the rest of the teams should deal with. So you should definitely have time and, for too long, load testing was, how do you say, something you would save until the end because you lacked time. But, of course, it’s not the proper way to do it and you will have some problems that are way worse than just losing time, of course.

Federico:

Yeah, sure. Okay, thank you for that. I went to talk also about Gatling. I have a question related to what was the motivation to start with a new tool five years ago?

Paul-Henri:

Yeah.

Federico:

Why did you decide to choose to start with a new tool? What was missing in the market, in the current tooling that you could get five years ago?

Paul-Henri:

So before I met Stefan, who is my business associate and who also is the creator of Gatling, I need to explain history a little bit. He was an IT expert and was the CTO of an IT consulting firm and he was building projects for large companies. 

He needed to do some load testing for his projects and what he used at that time were very famous tools in the load testing industry, but he wasn’t quite satisfied with these tools for various reasons.

First of all, as I said, that kind of approach, when you [load] test at the end, isn’t sufficient and he wanted to have a tool where you don’t just test at the end of a project, but you test at the very beginning of the project. That’s where you can gain time. You can gain a lot of things by starting to test early.

Also, he wanted to have a tool that could be implemented in all kinds of development projects and this is how he came up with the [inaudible 00:14:36] approach so that any developer that was starting to develop an application could easily start to implement these kinds of tests.

What he saw mainly in the organization he was working for was a testing team that was very centralized and what you needed to do at that time when you were a developer, you needed to develop your application and request internally to have a load testing campaign at the end of that. [inaudible 00:15:13] people came with reports and said, “So these are the response times of your applications.” 

But unfortunately, the developers didn’t have the opportunity to question the methodology or to understand what was tested so far. That was something Stefan wasn’t happy about.

So he said, okay, we should have a tool that all projects should use, that everyone could understand, everyone could question the methodology, everyone should be able to understand what is actually tested. Also, he wanted to have a tool that could be easily integrated in the development pipeline, that could plug with your continuous integration solution, that could plus with all the tools the developers were using, etcetera, etcetera.

And he managed actually with this approach to convince more and more development teams to start to implement that kind of strategy, so the load test even though the top management or the head of the project didn’t plan to have any load tests. 

Because what happens when you have a performance issue, people turn back to the developers and say, “Hey, what are those kinds of issues? Solve them immediately!”

So that’s where developers were forced to do load testing, even though the top management or the head of the project didn’t ask them to do it, and Gatling was a good way for them to start to implement this very easily.

Federico:

So they were the ones convinced that they wanted to start implementing the load tests right away. I understand.

Paul-Henri:

Yes, exactly.

One of the advantages of this approach is that developers who know how the application is built have to craft the [load] test. 

It doesn’t mean they are in charge of running the test, but they have to build them. 

They have to put them in a repository to track the changes, to do some peer review. Everyone joining the project should be able to understand what the load tests are truly testing. So that was the first thing he actually solved.

This is when I met him, actually at that time, he started to have a small community all over the world and I realized that it was a huge challenge for any kind of company; performance. So we were back in 2014 and I realized also that maybe the project lacked a bit of collaboration. 

So it was really developer-oriented whereas [load testing] is everyone’s matter actually. It’s a matter of head of projects, of CTOs, of testers, of ops… of everyone in an IT department.

So when I met him, I convinced him that there was way more to do regarding load testing and that we should build a company. So this is what we did and what we did first is, well, try to understand how the project was used by our users, trying to understand what they need, what kind of things they were ready to pay for. Because, of course, we needed money to continue the project and finance all the research and development.

And we realized what everyone was trying to build on top of Gatling was a collaborative tool or a new tool where everyone could connect, see all the testing campaigns that were playing, all the reports, the evolution of your different KPIs. You also need to manage your infrastructure and that because, as you know, you need a lot of infrastructure to do a load test.

So we came up with the idea of an enterprise tool on top of it that could manage all these aspects and really where all people in every IT department could connect and interact. So this is how we came up actually with that.

Federico:

Very interesting story. Thank you for sharing it. Also, now I understand how you ended up working in a startup, working in performance testing specifically.

Paul-Henri:

Yeah, people are always a bit surprised because I don’t I’m not a developer. I don’t have any IT background. I graduated from school but I wanted to work, like everyone, in innovation, in startups, but I was a bit sick of hearing about innovation every day. And when you dig a little bit in different startups, you see there isn’t that much in terms of innovation.

Paul-Henri:

When I met Stefan, when I saw what he has built so far, I said, “Hey, okay, that’s what I want to do.” So we are right in the middle of innovation, I mean the hard stuff, you know. It was quite obvious so this is what I wanted to do. This is what I wanted to develop with him. So this is how it started.

Federico:

I can see that there are many ideas to work on. Perfect! Okay… Let’s talk about more personal things, I would say.

Paul-Henri:

Yeah, sure.

Federico:

I truly believe that, in order to improve something in your life or your career, what you have to do is modify your habits. That’s why I want to ask you what habits do you have to suggest listeners try out or maybe avoid. Do you have anything you want to share?

Paul-Henri:

I would say one habit that is very important that I didn’t do for a long time is to talk on a regular basis with everyone in your company for a simple thing, not to talk about business, about KPIs or about projects or things like that, but just to detect if there is any problem for your employees, if they’re having some issue that impacts the work, if they have any issues with someone else in the company, before it becomes a real problem. 

It requires a lot of time to spend time with everyone, to spend time to reach out, to talk about everything but you will gain a lot of time by doing so instead of dealing with a problem when it’s huge.

It’s exactly the same thing I explained earlier with load testing.

“If you do load tests early, you should be able to deal with problems quickly. That’s exactly the same thing when you talk to everyone. It’s your job as a CEO to detect problems and deal with them quickly before they become problematic.“

PAUL-HENRI PILLET

That’s something a lot of CEOs told me to do. I didn’t do it at first but I realized, after a few years, that it was really important to do it. It takes time, but it’s totally worth it.

Federico:

Yeah. I love it because it’s taking care of your most important asset which is your team.

Paul-Henri:

Right.

Federico:

Talking about suggestions, do you have any suggestions for reading? Do you read software testing books or any other type of book? Do you have a suggestion?

Paul-Henri:

I love to get feedback from people who build companies because everyone has different problems but, at the same time, there’s some problem that you find in every project and you know you’re going to face them, you know there are solutions but sometimes you just can’t help it. You’re heading towards these problems and you just don’t deal with it. But it’s always important to know that you’re going to face that kind of problem because you will be able to react more rapidly.

There is one book I always remember with a very interesting anecdote is a book called “The Hard Thing About Hard Things“. It was by Ben Horowitz I think. 

There is a story which I remember… It was a long time ago I read that book, but there is this specific story where they’re about to lose their biggest customer and they don’t understand why and it’s very critical for the business. If they lose it, they will be forced to close the business and so they start to investigate. They don’t have much time and they start to investigate to understand what is blocking within that customer and they don’t find it. They finally find someone who is blocking the decision-making and they are discussing it with him. They understand what he’s looking for, what are his challenges within his own company and they say, “Okay, we know what we need to do. Let’s do that. Let’s work day and night to try to solve this and to offer a solution for that.”

And they manage to get the customer back. I think it’s really, really important because you’ve got everything regarding entrepreneurship there. As a CEO, it’s not always business as usual so sometimes we’ve got some situations we need to deal with and it requires human skills. It requires patience but it also requires reactivity so you need to act fast. I love that story because you’ve got everything in it. So yeah, that was one of the books I think it’s worth reading.

Federico:

Okay! I really like it. Also, I really like the story behind why you choose that book. Thank you.

Okay, thank you so much for your time, Henri, and in order to wrap up the interview, is there anything you would like to invite our listeners to do or maybe to download some tool or try something new?

Paul-Henri:

Just one thing. We have, of course, this enterprise tool [Gatling Frontline], and I’d like to say to our community or even to people who are not using Gatling, feel free to contact us and have a try with that tool.

First of all, because it’s free and you can easily switch back to the open source version. And we’re really confident because this is what we are seeing so far, that people are really convinced by what we have built so far. So really, contact us, have a try. And if you’re not satisfied, it’s not an issue. You can easily switch back. We wouldn’t mind actually. 

We always love to get feedback even if it’s negative.

Federico:

Okay, cool. Thank you so much for your time.

Paul-Henri:

Thank you very much.

Federico:

I really appreciate it. What else? Stay safe.

Paul-Henri:

Yeah, you too.

Federico:

And see you soon.

Paul-Henri:

See you soon. Thank you very much.

Recommended for You

Gatling Tool Review for Performance Testing (Written in Scala)

Gatling vs JMeter: Our Findings