Can you run performance tests in production?

Imagine that we have to run a test that we’ve already designed with a good load scenario, test cases, test data, all automated and prepared in the load simulation tool that we want to use, BUT we don’t have a testing infrastructure that compares to that of production. This usually happens when the production infrastructure is a large server, such as an AS400 or any other mainframe, a cluster composed of multiple machines, or a complex configuration, making it expensive to prepare an exclusive testing environment.

Where Should We Execute the Performance Test?

Surely the development server, or the one intended for functional testing, may not have the features that the production server has. What are our options in this case?

Well, the best option is to do performance testing in production!

We’ve been asked many times if you can run performance testing in production. What is the problem? Well, we assume many risks. What if you run a test of this kind on a system but then it affects the other systems that are in production? More we’ve been told that the AS400 does not fail, but this is not so, as it has happened to us more than once. That’s the main risk.

Moreover, a major problem is that the results we get are impacted by other applications that are in production on the same server.

However, this approach has a great advantage: we ensure that the test infrastructure is similar to that of production because they’re exactly the same! In this, we are including everything that relates to hardware, database software (including versions of each component, their configuration, etc.) and other systems that have to coexist in production (from antivirus software to the company’s other information systems).

Therefore, what we thought was a serious problem, is actually the ideal situation!

Conclusion

Do initial performance testing on another machine wherever you can to minimize the risk of finding serious problems that affect the performance of applications that are already in production. Try to solve the problems arising there under the load and then move to production. Whoever controls the test must have looked over the basic indicators: CPU, memory, disk and network, and be clear about what thresholds are no longer acceptable. Once those levels are reached, stop the test.

We could also consider running performance tests at times when there is less of a load, like at night or during weekends. The problem with this is that more than one person is needed to operate the simulation tool. If you really want to get good results from a performance test, it is necessary to have experts in infrastructure, someone from the development team, etc.

Have you faced this situation before? What approach did you take?

Recommended for You

Why Performance Testing is Necessary
How to Make a Performance Test Plan