Using JMeter 5? Understand this key difference that will impact how you set up your tests…

At Abstracta, we’ve been experimenting recently with JMeter 4 and 5 (the latest versions to date) and we found a very important difference in the way they behave and it’s important to know it in order to simulate load the way you’d like. 

Basically, the connections in JMeter 4 are kept open between iterations and in JMeter 5 they are completely restored, which leads to two implications that will drastically change test results: 

On one hand, JMeter 5 requires more resources to execute, and on the other, the registered response time will be longer than when the connection is reused (as it is in JMeter 4). 

Both consequences are due to the opening of a new connection requiring more resources, which takes more time than using one that’s open already.

In this post, written with the help of my Abstracta colleagues, we’ll tell you how we reached this conclusion and discuss the implications of it for making the decision of how to model your test to simulate your system’s users in the best way. 

Experimenting with JMeter 5

When executing tests with the latest available version of JMeter (5.1), we found that with about 30 virtual users, the use of the load generator CPU was already saturated, reaching 100% for almost the entire execution of the test. We could also see that the response times were high, and consequently, the number of requests per second was low.

To understand why this was happening, after analyzing different things, we wanted to try a previous version to verify if this also was the case. When executing the same test with JMeter 4.0, we found that we had considerably less response time, more requests per second and also less CPU usage than with JMeter 5.1. After a little research, we found that the main difference in this behavior was located in the connection time and when we reviewed the documentation of the changes in JMeter, we came across this:

“Since JMeter 5.0, when using default HC4 Implementation, JMeter will reset HTTP state (SSL State + Connections) on each thread group iteration. If you don’t want this behavior, set httpclient.reset_state_on_thread_group_iteration = false ”

(Source: https://jmeter.apache.org/changes_history.html)

In short: the connection in JMeter 4 is kept open between iterations and not in version 5. This causes the same test in JMeter 5 to have longer response times, lower TPS and need more resources to execute. 

Seeing the Difference

Below we can see a comparative table of the results obtained when running the tests with different versions of both JMeter and Java.

The script that was used for this test (origDemoblaze.jmx) was created with JMeter version 5.1 and simply consists of a single GET request to the address, http://dev.demoblaze.com, followed by a 300 millisecond timer to generate a minimum pause between requests.

Below is the YAML code of the Taurus script from which each JMeter script is generated, in case you also wanted to experiment with it:

In the table, you can easily see that the results obtained using version JMeter 4 consumed considerably less of the load generators’ resources and also managed to produce more requests per second and less response time.

For this second comparison table, we remove the 300 millisecond timer and observed the following behaviors:

Despite not seeing significant improvement in the resource consumption of the load generator this time, we noted that with the use of JMeter 4, the number of requests per second is significantly higher and response times are reduced compared to those obtained with version 5.

How to Simulate Your Desired Behavior in JMeter 5

So, is this big difference something positive or negative? Is the default behavior of JMeter 5 correct? 

It all depends on what we are trying to simulate in each case. 

Mainly, we have to think about what is it that simulates every thread we execute with JMeter. A thread will execute a script many times until the test is completed. There are two options, that a thread simulates a user performing the same flow of actions many times (those automated in the script) or that a thread simulates different users who access the system one after another, performing the same flow. 

Keep in mind that every browser reuses connections. If the users who are going to access the system are going to perform the same task many times, we really want the connections to be reused (as they are in JMeter 4). If instead we are going to simulate many users, one after another, it’s best not to reuse the connections (the way JMeter 5 does). 

The idea of this change is that each iteration is like starting from scratch, as if it were another user, so one user = one iteration of a thread. In this way, the threads are simply a control of how many active concurrent users we can have.

Anyway, with JMeter 5 we can have the same behavior we had in version 4 using a loop controller within our script or in the newest version of JMeter, activate the checkbox in the Thread Group panel that says “Same user on each iteration”. Below we also leave you a new script with a loop controller added so you can experiment with it:

The counterpart of this new reset operation “All connection states for each thread”, limits the number of virtual users that can be run by the same load generator since it consumes more resources from them. This means having to use more load generators in some cases, which increases the cost of execution.

Closing

In conclusion, when using JMeter we must be clear about 1) the scenario we want to simulate and 2) the differences in behavior between one version and another in order to make the best decision. 

Picking wisely will help us to obtain results that are closer to reality and thus be able to better analyze the performance risks we face. 

Recommended for You

When is the Best Time to Start Performance Testing?
How to Quickly Set Up Test Automation in CI/CD