Don’t fall for these common performance testing fallacies

It’s always interesting to find out the many ways in which we can be wrong. Here we want to point out the various software performance testing fallacies that we have seen that have led to the use of poor methods, which end up costing a lot more money down the road. This is our first post on the topic, for part two, click here.

In his book, “Perfect Software and Other Illusions About Testing,” Jerry Weinberg explained a number of fallacies regarding testing in general. In this post, we’ll cover five performance testing fallacies. 

The Planning Fallacy

We often think that performance tests only take place at the end of a development project, just before rollout, in order to do some fine-tuning to make sure everything goes smooth. That is how performance testing is seen as a solution to performance problems. But, in fact, it’s about detecting and anticipating problems in order to start working on their solutions. The greatest problem is that when we consider performance testing only at the end of the project, we will end up encountering very serious problems whose solutions will involve higher costs. It would be best to consider performance, from the early stages of development. One should carry out intermediate tests in order to detect the most important problems that might arise.

The “Just Add More Hardware” Fallacy

It’s typical to hear that performance testing is not necessary because any problems detected may be solved by simply adding more hardware like additional servers, memory, etc. Consider the case of a memory leak. If we add more memory, we might keep the server active for five hours instead of three, but we won’t be solving the problem. It also doesn’t make any sense either to increase infrastructure costs when we can be more effective with what we already have and reduce fixed costs in the long run. In short, adding more hardware is not a good substitute for performance testing.

The Testing Environment Fallacy

There is another hardware fallacy asserting that we can perform tests in an environment that may or may not resemble the actual production environment. For example, testing for a client on Windows assuming that the application will function perfectly for another client who will install the system in Linux. We must make sure to test in an environment as similar to the production environment as possible. There are many elements from the environment that have an effect on a system’s performance. Some of these elements include hardware components, settings of the operating system, and the rest of the applications executed at the same time.

Even the database is an important aspect of the performance testing environment. Some think that performance tests may be carried out with a test database, but in employing one, problems with SQL queries might go unnoticed. As a result, if we have a database with thousands of records, the SQL response time will not have been optimized and would surely bring along tremendous issues. This is why it’s important to keep the test environment as similar to the actual environment as possible.

The Comparison Fallacy

It’s one thing to assume that you can use a performance testing environment that does not resemble the actual production environment, but it’s another to make conclusions about one environment based on another. We should never extrapolate any results. For instance, you cannot duplicate servers to duplicate speed. Neither can you simply increase memory to increase the number of users supported. These assertions are simply mistaken. In general, there are numerous elements exerting an impact on the overall performance. The chain breaks at the weakest link, so if we improve two or three links, the rest will continue to be equally fragile. In other words, if we increase some of the elements that restrict a system’s performance, then the bottleneck will become another element along the chain. The only way to make sure is to keep on testing performance.

Extrapolating in the other direction is not valid either in performance testing. Imagine the case of a client with 1,000 users executing with an AS400 functioning perfectly. We cannot consider the minimum hardware necessary to provide support to ten users. We must try it to verify it through testing.

The Thorough Testing Fallacy

Thinking that one performance test will prevent all problems in itself is a problem. When we go about performance testing, we intend (due to time and resource restrictions) to detect the riskiest problems that may have the greatest negative impact. We usually limit the number of test cases (usually, to no more than 15) because it is VERY costly to carry out a performance test including all functionalities, alternative flows, data, etc. This means that there will always be situations that go untested that will produce, for instance, some blocking in the database, or response times longer than acceptable. The main thing is to cover the main cases, the riskiest, etc. Every time a problem is detected, we must try to apply that solution to each part of the system where it could have an impact. For example, if we detect that the database connections are managed inappropriately in the functionalities being tested, then once a solution is found, it should be applied in every point where connections are involved. Solutions are often global, such as the configuration of a pool’s size, or the memory assigned in the Java Virtual Machine (JVM).

Another valid approach that proves reassuring when it comes to performance testing is the monitoring of the system under production conditions, in order to detect any problems that might have arisen because they are outside the scope of the tests, so that they may be corrected promptly. Remember, just by running a performance test, you are not always completely clear of any possible problems, but there are several ways to ensure that you minimize that risk.

What fallacies have you heard of or dealt with? Comment below!

For more performance testing fallacies, continue on to part two.

Recommended for You

Top 10 Factors Impacting Application Performance
Cloud Performance Challenges