How do we select good assertions?
For those of us who work with JMeter and performance tests, we know the importance of adding validations to all HTTP Requests. Validations are not just a way to confirm that the server is responding as expected, but they also serve to verify that the script we are developing is acting as it should, meaning, it is simulating the user behavior as it was designed.
For example, if running our script on the system implies that a record is created in the database, we have to ensure that after running the script, the record has indeed been created and that the script doesn’t say everything went smoothly if it didn’t actually create it. That would be an example of a bad choice of an assertion or validation of our requests.
We want to avoid false negatives and false positives. Here’s a review of what those mean:
- False Positive: The test says there is an error when there really isn’t one and then we lose time looking for the cause of the error until we realize that the error was just in the test, or in the environment, or the data, etc.
- False Negative: There are errors but they are not found. Since they were not found, we believe that these features have been fully covered and they are bug-free although they are not.
If there is an error, it’s important that the script fails. Conversely, if it goes well, the script has to confirm that it went well and we have to be able to trust it.
How do we know which response assertions to add to our scripts? The reality is that I have not found a guide on how to choose these, and it always stumps me when I am teaching JMeter to someone and they ask me how do I know what to validate in the response of an HTTP request?
As I mentioned in my checklist, 8 Ways to Improve JMeter Scripts, just validating a response code is not enough, because we can lose other errors that we need to capture in order to act accordingly.
Therefore, today I am sharing with you the methods and criteria for selecting which JMeter response assertions to make when validating the responses of the server.
1. Validate the Login
Typically, after authenticating a system, a “Welcome @username” message appears and our name appears somewhere on the screen, indicating that the login was successful. In these cases, we have an excellent element to validate.
Let’s see this with an example:
1) I log in with the user’s credentials.
2) I successfully enter the site and it shows the user name on the screen.
3) I search for how the element appears in the response HTML.
4) I validate it in JMeter.
Recall that in JMeter, we can specify a variable in the validations, then, in cases where the username is parameterized (read from a CSV file or defined as a user variable), one can verify the text ${UserName} and JMeter will replace the variable value.
2. Validate Window Content
Many other times requests return a page in which you can choose text/titles that should come in the HTML response. For example, if we access the JMeter site, we can see the following:
We know that if this page loads correctly, the title “What can I do with it?” should appear in the response to the request. If it doesn’t appear, then you know you have an error, so it’s a good assertion to add as a validation that we are getting the expected page.
The challenge in this type of validation is to verify content that we know that, if there is an error, then that text does not come in the response. We can rely on developers to provide us with this kind of information.
3. Consider the Next Steps in the Script
If we know that the next action to take is clicking a button X, or accessing the link Z, or selecting in the combo Option B, a very good strategy is to validate that the element on which we operate in the next step, appears in the response of the request. For example, if after loading the JMeter site, I want to select the option “Download Releases” I can validate this text and I assure the flow continues.
For this, I capture the response from the server verifying that it includes the option I want to check:
And then I create the validation in JMeter:
4. Validate Values That Need to be Extracted
There are times when we obtain variables from the response of an HTTP request through the use of regular expressions (Add -> Post Processors -> Regular Expression Extractor) with the intention to use that variable in a subsequent request.
We can then validate that the variable appears in the server response.
For this, we can make use of validations that use regular expressions.
For example:
Thus, if the regular expression doesn’t match in the response, we will get a validation error.
5. Take Special Care with Special Characters
Be careful when writing the assertion in JMeter because the text is not always given in the same format that is displayed. For example, to validate the title Apache JMeter ™, I cannot write it verbatim because the server response comes as Apache JMeter & trade; and not the other way as seen on the screen. If you do not use the text as it appears in the HTML, the validation (and regular expressions) will not find it.
6. Use a Proper Amount of Assertions
JMeter allows us to add as many assertions as we want, so that each of the assertions that were named above may be accompanied by other assertions such as the titles of the HTML responses (<title> … </ title> ), in order to add strength to the validation. When adding more assertions, we have to be careful that the script is not too loaded, which is why it’s not advisable to add more than two assertions per request, so we must ensure that the ones we choose are determining factors in defining the correctness of the response.
We hope that this list will be useful for you when you add assertions. Can you share any other ideas you may have?
Are you testing or developing a mobile app? Check out Apptim, a free tool to fully test your mobile app and analyze its performance, preventing any issues from going live to your users.
Recommended for You
How to Make a Performance Test Plan
Top 14 Performance Testing and APM Tools
Quality Sense Podcast: Andreas “Andi” Grabner – Introduction to Keptn
In this Quality Sense episode, host, Federico Toledo chats with Dynatrace’s Andreas aka “Andi” Grabner. From Austria, he has more than 20 years of experience in the field and continuously shares his knowledge about performance engineering especially through his podcast called Pure Performance. During the…
How to Execute a Performance Test With Fewer Problems
Executing performance tests more effectively and efficiently A thought by Scott Barber with which I fully agree reflects part of what I want to discuss here: “Only performance testing at the conclusion of system or functional testing is like ordering a diagnostic blood test after the…
Abstracta
Hi Leticia,
I have one question, I have one request with the payloads, but I am getting at least two different response code for same payloads.
Scenario 1 : If i am the new user going to register after successful registration i got 201 or 200 successful message.
Scenario 2 : If i again request for the user registration after successful validation from the server I am getting 409 conflict.
with response body, user already registered.
I wanted to handle both the response code and the response body in one assertion.
Please help me
Thanks
Mohammad Shahnawaz
Hi, I have a question that can we able to use .xml file to access the number of user details in the Jmeter script?
Hi Kani, The tag you should check to get the number of users in the xml is: name=”ThreadGroup.num_threads”.
Hope this solve your question.
Bests!
Hi Leticia, How can we assert java application log through jmeter.
Hi could you please help me. I have made a request to a website and after iteration in a loop controller the request is repeated. I apply assertion to request checking some words on the page like logout and user name, but occasionally server responds with sub samplers with codes 302 and 304 not modified and the result of main request is empty it causes a failed result in assertion. How should i handle this case correctly? What assertion should i add insted? Thank you.
Hi,
First of all, I recommend you to review why the server is responding in that way. Is it that behavior you are expecting?
If it’s ok, you could use OR option into the assertion sampler configuration and assert response code 200 OR 30 (contains or substring option). Hope this help! Try and let me know.
Thank you! I was recommended to use groovy assertion:
def code = prev.getResponseCode()
if (code.equals(‘200’)) {
if (!prev.getResponseDataAsString().contains(‘some words’)) {
AssertionResult.setFailure(true)
AssertionResult.setFailureMessage(‘Required words were not found’)
}
}
else if (code.equals(‘302’) || code.equals(‘304’)) {
// do nothing
}
else { // response code is not 200, 302 or 304 – fail the request
AssertionResult.setFailure(true)
}
Now I have another problem:
In next iteration in a loop controller the response result is empty due to 304 code (not modified) and an extractor which worked well before is not able to retrieve needed information because of response is empty (main sample and subsamples results are empty but at first iteration everything was ok response code 200 and html code was received). I was recommended to apply an extractor to a main sample and subsamples but it doesn’t work. Have you any ideas?
Hi Youssef, you can try with Response assertion -> Response headers and use the content length to validate the ideal size (image quality) and you can also use a Size assertion if you need to be more flexible with the size. Hope it helps.
@ Leticia Almeida: sorry for interrupt, I need to measure an Image Quality after some processing using Jmeter
Thank you Leticia for the suggestions, I Will go through.
1. Thank you Leticia for your quick response , what if I have 5different response types to check but I don’t want to add 5 response assertions,rather put in one response assertion. If there are many to check,how feasible it is to write those assertions for each….please suggest, Thank you .
2. Another question, I have a requirement to add total no of pass & fail requests and show in output file, so how to put that logic & where I hv to write & how to write User defined variables into output file, I mean how to generate customized report in jmeter….please any suggestions, Thank you
1- In my opinion, I think that adding two of those five assertions is ok. But if you really need to assert five different conditions you don’t have another way.
2- You could use listeners in order to log your pass/fail requests. Logging user variables is a bit different, for doing that, I suggest you read this post: http://stackoverflow.com/questions/8352281/write-extracted-data-to-a-file-using-jmeter.
Hope this helps!
Hello, nicely explained.
I have a question, how can I check multiple response types using assertions,and I don’t want to create as many assertions I need, for e.g. In Response Assertion , we have Response data, response message, body text etc at a time, I can check only one type of response in one assertion, but what if I want to check multiple response types for the same request, how can we test such scenarios …please help.
Thank you .
Hi Saritha,
First of all, thank you for your question!
The answer is yes, you can add differents assertions for the same request. You only need to add another response assertion sampler to your request and set it as you want.
Hope this helps!
Hi, I have a question about how to use MD5Hex assertion to verify the downloaded file:
I want to verify the downloaded file’s checksum, I checked ‘save response as MD5 hash” in the http request sampler, and then I added a ‘MD5hex assertion” with the md5hex as the result calculated by my Java application using the same file. But the two checksum is not the same.
I wonder what’s the reason. Is the response not the response body?
Thanks.
[…] JMeter Response Assertions: How to know what to validate in an HTTP Response Request […]
[…] The idea is to verify that the test case realizes the mistake and that’s why we try to make it fail with these alterations. Anyway, what we could at least do is think about what happens if the software fails at this point, will this test case notice it, or should we add some other validation? […]
Nice article but IMHO it is important to note that those assertions may add performance overhead
Explained Response Assertion nicely. Expect more articles on other assertions with example too specially BeanShell Assertion, MD5Hex Assertion and XML Schema Assertion if have any idea.
Hi Aravind.
First of all I’m glad you enjoyed the article.
About the others assertions, those are great topics. We could in the near future write something about that.
Thanks very much for your comment and the idea!
In this article you mentioned it is advisable to use 2 assertions per request. is there any where defined on this or based on experience.
Hello!
It’s just based on my experience in order to avoid heavy processing along the execution.
Thanks for asking! All questions are welcome.