Have you heard the hype around k6 yet?
Yes, yet another open source performance testing tool is making waves. In this post, I’ll share my k6 review in the hopes that it may help you decide if you should also pay attention to it.
k6 is a developer-centric, free, open-source load testing tool that’s been receiving a lot of attention lately. In 2016, the k6 team set out to build a great open source load testing tool (formerly known as LoadImpact) and has been pushing out updates and improvements ever since. In early 2020, the tool reached two million test runs per month, so it’s fair to say that its user base is already well established and if this growth trend continues, there should be even more to come.
The Mission Behind k6
Next is the fact that k6 has really good documentation (https://k6.io/docs/), however, there are certain things that you may find hidden where you’d least expect and it’s lacking in some specific examples for how to do things.
Another remarkable thing is the number of integrations that k6 manages to handle. You can convert a HAR file to a k6 script, which means you can record a workflow in your browser or in a web debugging proxy tool such as Fiddler or Charles Proxy, save it, and then convert it to a ready-to-go k6 script.
I also find it impressive how you can convert JMeter scripts, Postman collections and even create a script from an API’s Swagger (I haven’t yet tried any of these options myself but they’re well documented on its website). As expected, k6 also easily integrates with the most common CI/CD tools such as Jenkins, GitLabCI, GitHub Actions, CircleCI, and more.
Finally, its results reporting is quite good. You can design custom metrics, checks (aka assertions, for those coming from JMeter), create an output of the test result in JSON or integrate it with a number of different data visualization tools such as DataDog, Grafana, or Kafka.
The Not So Good
If you’re coming from other popular tools such as JMeter or Gatling, you might find the way of doing certain things more difficult or somewhat “hacky.” I’ll give you two examples of this that I came across when trying to create a script that simulated some simple user behavior interacting with a dummy e-commerce website.
First, if you try to group your requests (this might not be the only way to do it, but it was the way I found out) but want to say, get a value from a response to one of the requests in group A and then use it in group B, you have to declare the variable in the supergroup as undefined, then set it in group A and then use it in group B. In JMeter, this is easier to achieve and doesn’t require as much trial-and-error.
Finally, there is no easy way to perform general checks or set acceptance criteria for the whole test. What I mean by this is that when using different groups as I did, you can’t define an error percentage or response time threshold for all of the requests inside each group from the “master” group. You can either go ahead and repeat the same checks in each of the subgroups or define custom rates which will act as general checks, but it would be nice to have that feature readily available rather than having to go through the trouble of defining custom checks. This could also be accomplished if Taurus offered a k6 runner, which is not yet implemented.
Comparing k6 to Gatling
For k6, I think the fairest comparison you could make against similar tools is with Gatling.
Ease of Use
Parameterization is easier with k6. In Gatling, if you want to change, for example, the number of VUs between runs, you first need to declare a variable in your script, which will then take the value from the maven parameters passed when running the script. However, in k6, you can have a hardcoded value for VUs in your script, run it, then run it again passing a flag and overwrite that value, without needing to touch the script.
Maintenance is easy using both as they’re designed with the “load tests as code” mindset and simulations should be stored in some sort of version control system. Gatling also supports a wide variety of protocols either natively or with help from its users’ communities while k6 only supports HTTP1.1 / 2 and WebSockets.
When it comes to reporting, I’d say Gatling is the winner today since k6’s built-in reporting is not quite as advanced. One of Gatling’s best features is its automatic report generation, which creates a really nice webpage that allows you to clearly view and navigate the vast amounts of data logged by the tool. It’s even clearer when executing, while k6 displays little to no information on the terminal when running a simulation until it ends, Gatling has some real-time reporting. In any case, these tools can be integrated with some other open source solution to improve the real-time reporting, such as ELK or Grafana. This capability is important, for example, to cut a test run short if you see failures are happening. However, if integrating k6 with other tools for better data visualization is an option, you can produce really great reports that way.
As for distributed load testing, it becomes a bit muddier. Both tools offer a paid, cloud-based version that better support reporting and distributed executions. Gatling however, also offers a free shell script which allows you to distribute your simulation amongst as many machines as you’d like. This script does require some previous configurations in all of the generators though.
On the other hand, the k6 team states that its tool is more performant than its competitors’, thus reducing the need for distributed load testing since you can supposedly generate serious load on a single generator, although we couldn’t check this claim (the test sites we were using started rejecting requests when the load got heavy). It seems to me though that this argument falls flat when, for example, you want to distribute the load between different geographic areas in order to simulate more real-world scenarios.
The Final Verdict
If you’ve made it this far through my k6 review, thanks for reading—and I’ve got to say, I like k6, far more than I thought I would. I believe it has a lot of potential and it’s clear that the k6 team is highly involved, focused on improving it so that it’ll continue growing its user base and features.
I encourage anyone interested in new load testing tools to take a crack at k6 and let me know what you think about it in the comments!
Recommended for You
Introduction to Taurus: An Alternative to JMeter
JMeter struggles? Taurus to the rescue! For those of you who know JMeter (the most popular open source tool for testing performance), your first thought was probably not “What a user-friendly software program! It’s so clear what this script does!” Taurus is here to fix that…