Should we test performance throughout development or at the end? – Agile vs. Waterfall approaches

When taking into account the performance of existing systems or ones built from scratch, teams have to determine at what point in the development process they are going to benefit most from running performance tests. I’ve spoken about this topic at a couple of conferences, including the CMG imPACt Conference in La Jolla alongside Sofía Palamarchuk, and I thought, “Why not summarize the talk in a post?” So, the purpose of this post is to answer the question:

Should we start performance testing at the beginning, alongside development (taking the agile approach) or at the end (the waterfall approach)?

First, a quick overview of what performance testing implies:

A computer’s performance is characterized by the amount of useful work accomplished by a computer system compared to the time spent and resources used. Remember, performance doesn’t only refer to speed. For example, a system that is very fast that uses 100% of the CPU is not performant. Therefore, it is important that we check both the user experience (the perceived response time, or the velocity) and how stressed the servers are. Furthermore, if we only pay attention to response times, we could only be seeing symptoms when what we really want to find are the underlying root causes in order to identify those bottlenecks and then the ways in which we could improve.

Now, back to the question at hand: Should we take the waterfall or agile performance testing approach?

To clarify what we mean by the two, the agile approach is when we start performance testing at the beginning of the development process and continue with it along with the whole evolution of the application.

The waterfall approach is when we leave all the performance testing activities for the end of development, as acceptance testing, checking that the system performs as needed.

Let’s take a look at what they entail and then the advantages and disadvantages of each.

The Waterfall Approach

In this approach, we usually wait until the end of development to start testing. The performance tests take the form of acceptance testing and if the criteria are met, the system is ready to go into production. This involves simulating the expected load scenario.

Pros

• It is easier to plan and assign resources for performance testing since you are only doing it for a designated period of time.
• Typically, you try to use a test environment as similar to that of production as possible, which is beneficial because it is more realistic.
• It allows for the focus on specific characteristics to test (like X amount of functionalities under a specific context).

Cons

• While it’s great that we are testing in a similar environment to that of production, at the same time it can be difficult to obtain the infrastructure for an exclusive usage for the tests (you need to isolate the SUT in order to have trustworthy results.
• The cost of making architectural changes toward the end of development (if testing discovers that it’s necessary) is high.
• There is a high risk that comes with waiting to assure performance at the very end because you never know how much work you will have in front of you to go back and fix things in order to reach your performance goals.

The Agile Approach

The agile approach to performance testing entails starting testing from the very beginning with unit tests. It is key to have a continuous integration environment in place and what happens is that instead of merely carrying out performance testing, we carry out performance engineering.

Pros

• Minimize risk.
• Obtain early, constant feedback.
• Learn about best practices as you go and over time, continuously improve. When you start testing early, if you do something wrong, you have time to catch your mistake and avoid making it again. It’s great for reducing the likelihood of spreading bad practices all over the system.
• Facilitate continuous integration.

Cons

• It requires more automation effort in writing and maintaining scripts.
• Problems may arise if you automate too little or too much at certain levels. For example, it’s best to automate as many unit performance tests as possible, have some at the API level, and only automate the most critical test cases at the GUI level. This is in accordance with Michael Cohn’s automation pyramid idea but applied to performance. Be aware that you will have to ponder what a performance unit test is in your case.
• Sometimes teams fail to recognize that it’s a fallacy that if you test components separately, the system will work properly. That’s not necessarily true. You need to test the components individually and then, test them working together in order to achieve the best results.

When choosing between these two approaches, it is first important to take stock of the people, technology, and processes with which you have to work. It is important to have testers with both the proper soft and hard skills for performance testing. You also have to take into consideration which tools to use when load testing (ie, JMeter, BlazeMeter, Gatling, etc.) and monitoring both on the server side (ie, New Relic, NMON, perfmon, etc.) as well as the client side (with tools like Apptim, Page Speed, and Yslow,). Processes include test design, test automation, test execution, and measurement. When coming up with an execution plan, we recommend testing against a baseline and then use an iterative, incremental approach (I’ll write a follow-up post on what that means soon!).

So, which approach is right for you? Well, it depends what your desired outcome is.

When to Go with the Waterfall Approach

You may want to run a load simulation at the end when:

• You need to verify that your existing system supports a certain load.
• Your customers require evidence that your system meets a certain standard for performance (for example, if your client is a bank and it wants to make sure that its online banking system can support 100,000 daily users).
• If you suspect that certain tuning is required for the specific context where the application will run.

When to Go with the Agile Approach

You may need to take this approach that entails performance engineering throughout development when:

• If you want to reduce risk and costs.
• You want to increase the team’s collective knowledge of performance engineering and monitoring since they learn about it during the whole process.
• When your goal is to follow a continuous integration scheme.

Can we definitively claim that one approach is better than the other in every instance?

No.

We need both approaches in different stages of our development cycle. We should start early by doing performance engineering and we also need to simulate load for acceptance testing.

And, in reality, the two approaches are not all that different. Both require the same use of people, technology, and processes, but they slightly vary depending on how far along in development you may be.

Choosing Between the Two in Real Life

At Abstracta, many of our clients come to us asking to take the waterfall approach, with the intention of running load simulations and acceptance tests before go-live of a new version of their system or after making a change to their architecture, etc. One instance in which we’ve done this was for a financial institution that had recently merged with another, and it needed to assure that after having doubled the number of accounts in its internet banking system, the performance wouldn’t suffer.

Other clients have taken the performance engineering route, such as the e-commerce giant, Shutterfly, who runs performance tests continuously. This enables them to have a continuous integration environment, releasing updates frequently in order to enhance the user experience without allowing performance degradations. Read more about Shutterfly’s continuous performance testing scheme here.

If you are looking for more support, check out our other performance articles or contact one of our consultants today to start performance testing.

What are your experiences with the two approaches?

Recommended for You

Why Performance Testing is Necessary

3 Metrics Every Performance Tester Should Know