Continuous Performance Testing: The Best Way to Test Performance in Continuous Integration

Which are the benefits of continuous performance testing? Why is it so crucial to achieving an efficient continuous integration pipeline? Find out in the second article of our saga entitled “Performance Testing In-Depth”, with the voice of Roger Abelenda and an interview with Andréi Guchin.

We believe that an efficient continuous integration pipeline is the key to creating quality software and remaining competitive in today’s technological landscape. Why? Because continuous integration allows focusing on the changes delivered. Thus, it provides value to users sooner, trying ideas faster, avoiding regression problems, and streamlining software delivery.

Continuous integration is the practice of continually (ideally hourly or daily) integrating features and changes in an efficient way by automating tests that verify the proper functioning of the software after each change. The opposite would be to only integrate at precise points in time (sometimes after days, weeks, or months of development) when big features or modules are complete, followed by integration testing, usually through some manual or semi-manual process, and fixing any potential problems

There are many software quality assurance activities to focus on that will help you in your efforts to reach an efficient continuous integration pipeline.

Continuous integration tools allow for defining a sequence of tasks to be performed automatically and, thus, achieving the highest level of testing maturity according to our maturity model. Undoubtedly, Continuous Performance Testing is part of the core of this path. In fact, it is essential to make it possible.

Teams must be able to detect any errors of any kind as soon as possible. Practices like continuous performance testing go hand in hand with continuous integration (CI) and gain importance for Agile teams.

Continuous Performance Testing is a critical pillar for having reliable and highly performant web pages and applications of all kinds. And to be sure that its user experience doesn’t deteriorate over time with the introduction of new functionalities.

This article will focus on the benefits of Continuous Performance Testing and how we approach it at Abstracta. We interviewed Andréi Guchin, Performance Engineer and Lead of the Performance Hub at Abstracta, so you may have a clear picture of its importance in order to make progress in your testing maturity.

– What is Continuous Performance Testing?

In general terms, it is the method to evaluate the quality of a system in terms of its performance when working with a continuous integration approach. It consists of performing continuous tests, in the same way as those that verify the functionality of the application, with a focus on the performance of the application within the parameters considered acceptable for that system. Or at least that it has not degraded with respect to the previous version.

– Is this The Best Way to Test Performance?

This is the best way to do performance testing when working in continuous integration. Why? Because, unlike “traditional” performance testing, it focuses on measuring performance degradations between a system version and the previous one, instead of obtaining results for a specific version that can quickly become obsolete. 

In addition, following continuous integration best practices, this type of testing seeks to automate the execution of repetitive tests that would otherwise have to be run “manually”. This allows the tester to focus on more challenging or critical tasks that allow him to deploy his full human potential.

However, this does not mean that only continuous performance testing has to be performed, it is also necessary to perform traditional software testing for major releases where there is a significant change or that is expected to introduce some performance problem. All this taking into account the particular requirements of each release and the particular events expected.

– How to incorporate continuous performance testing into the dev pipeline?

First of all, it is necessary to define the testing strategy to be implemented. Among other things, it is necessary to focus on what functionalities are going to be tested, what data is needed, what load or what scenarios are going to be executed for each test, how and when they are going to be executed, what metrics are going to be obtained from the results, and how the results are going to be displayed.

You also have to choose the set of tools you’re going to use so that they meet the needs of the project and that the team feels comfortable using them. Then, we start with the automation of the tests and, as they are ready, they will be integrated into the execution pipeline.

It is important to point out that this whole process, which I summarize, takes time to analyze, work and go back and forth, like any other project. Sometimes it is not as straightforward as I am telling it and it is necessary to go back on some decisions, try other options, or remove previously defined things from the scope.

At the end of the day, beyond trying to follow a formal CI/CD performance testing methodology, the important thing is to adapt this methodology to my needs, or those of my business, and incorporate those practices that really add value.

– What are the benefits of continuous performance testing?

On one hand, there are the general benefits of performance testing, such as mitigating the risk of observing problems associated with the performance of the system once it is put into production. This reduces the likelihood of users being impacted by these problems, as well as the image of the developed product and the business being affected.

On the other hand, we have the specific benefits of performing this type of testing with a CI/CD approach: detecting deviations in the product as early as possible and correcting them in time. This reduces context-switching for developers. When working in sprints, the context often changes from one sprint to another. 

If bugs are not detected in time, then when the developer has to resume the task to correct it, he may be working on something else and it may be more difficult for him to focus on the problem again. 

Among the specific benefits of performance testing in CI/CD, it also allows for achieving greater collaboration between developers and performance testers, following the line of agile methodologies where interdisciplinary teams are promoted, and generating a performance benchmark for the most critical functionalities of the system.

– Are there any disadvantages in continuous performance testing?

Currently, the main disadvantage I see is that this type of testing does not 100% replace “traditional” performance testing. The information provided by the two approaches is complementary to each other. Then, there is the challenge of evaluating whether or not it makes sense to run “traditional” performance tests in a CI/CD scheme, but that is a separate discussion. 

– Is it more expensive or cheaper?

The answer to this question is a big “it depends”. It depends on the project, the company, the quality and stability demands of the application, and the established image of the company. It depends on the maturity of the team, how critical the quality performance level is for the system, and on the company. But above all, it depends on what is considered expensive and what is considered cheap. 

To give an example, someone may consider it expensive in terms of time and money to go to the doctor periodically for check-ups, but if we do not do it and in the long run we get sick of something complicated that we could have prevented with those consultations, it ends up being very expensive.

In general, in the short term, it could be said that it is more expensive since it initially requires an adequate investment. But in the long term, properly implemented, it is much cheaper and more profitable.

Why should you focus on performance testing to grow your business and reach scalability? Figure it out in this article.

Want to know about a success story where we conducted continuous performance testing? Learn about our work with Shutterfly, the leading manufacturer and digital retailer of high-quality personalized products and services, headquartered in the heart of Silicon Valley and acquired by Apollo Global Management for $2.7 billion upon its 20th anniversary.

Hosted by Abstracta, WOPR29 is just around the corner! Find more information here.

You may be interested in downloading The Ultimate Guide To Continuous Testing.

Follow us on Linkedin & Twitter to be part of our community!