Have you heard the hype around k6 yet?

Yes, yet another open source performance testing tool is making waves. In this post, I’ll share my k6 review in the hopes that it may help you decide if you should also pay attention to it.

Overview

k6 is a developer-centric, free, open-source load testing tool that’s been receiving a lot of attention lately. In 2016, the k6 team set out to build a great open source load testing tool (formerly known as LoadImpact) and has been pushing out updates and improvements ever since. In early 2020, the tool reached two million test runs per month, so it’s fair to say that its user base is already well established and if this growth trend continues, there should be even more to come.

The Mission Behind k6

In the industry, there has been a need for a load testing tool that is built for teams that everyone is able to use (from developers and testers to DevOps/SREs). The k6 team achieved this, democratizing load testing by building an application written in Go, (seeking to squeeze as much performance as possible out of the hardware) while the scripting is done using JavaScript. As you could imagine, in a world where JavaScript ranks as the number one most popular programming language by Stack Overflow for three years in a row, having a load testing tool whose scripting is based in it is a great way to lure developers into trying and ultimately, sticking with it.

The Good

First of all, as I previously mentioned, the fact that its scripting is based in JavaScript is a highlight of k6. Nowadays, full-stack JavaScript developers are more common, so they’re able to integrate a load testing tool into their workflow without having to adjust much or learn new languages/UIs. However, that doesn’t mean it’s hard to use if you don’t know JS: I’ve spent no more than 150 hours using JS, mostly working with WebdriverIO and Angular and I had no problems at all getting k6 to do whatever it was I wanted it to do. What I mean by this is that the barrier to entry is low, if there is any at all.

Next is the fact that k6 has really good documentation (https://k6.io/docs/), however, there are certain things that you may find hidden where you’d least expect and it’s lacking in some specific examples for how to do things. 

Another remarkable thing is the number of integrations that k6 manages to handle. You can convert a HAR file to a k6 script, which means you can record a workflow in your browser or in a web debugging proxy tool such as Fiddler or Charles Proxy, save it, and then convert it to a ready-to-go k6 script. 

I also find it impressive how you can convert JMeter scripts, Postman collections and even create a script from an API’s Swagger (I haven’t yet tried any of these options myself but they’re well documented on its website). As expected, k6 also easily integrates with the most common CI/CD tools such as Jenkins, GitLabCI, GitHub Actions, CircleCI, and more.

Finally, its results reporting is quite good. You can design custom metrics, checks (aka assertions, for those coming from JMeter), create an output of the test result in JSON or integrate it with a number of different data visualization tools such as DataDog, Grafana, or Kafka.

The Not So Good

If you’re coming from other popular tools such as JMeter or Gatling, you might find the way of doing certain things more difficult or somewhat “hacky.” I’ll give you two examples of this that I came across when trying to create a script that simulated some simple user behavior interacting with a dummy e-commerce website.

First, if you try to group your requests (this might not be the only way to do it, but it was the way I found out) but want to say, get a value from a response to one of the requests in group A and then use it in group B, you have to declare the variable in the supergroup as undefined, then set it in group A and then use it in group B. In JMeter, this is easier to achieve and doesn’t require as much trial-and-error.

Finally, there is no easy way to perform general checks or set acceptance criteria for the whole test. What I mean by this is that when using different groups as I did, you can’t define an error percentage or response time threshold for all of the requests inside each group from the “master” group. You can either go ahead and repeat the same checks in each of the subgroups or define custom rates which will act as general checks, but it would be nice to have that feature readily available rather than having to go through the trouble of defining custom checks. This could also be accomplished if Taurus offered a k6 runner, which is not yet implemented. 

Comparing k6 to Gatling 

For k6, I think the fairest comparison you could make against similar tools is with Gatling. 

If you aren’t familiar with Gatling, check out this review or listen to this podcast episode featuring its CEO, Paul-Henri Pillet. 

Ease of Use

For me, the barrier to entry is lower with k6 than Gatling. My reasons for this are the fact that JavaScript is more widespread than Scala and also because I found k6’s docs to be more helpful than those of Gatling.

Parameterization

Parameterization is easier with k6. In Gatling, if you want to change, for example, the number of VUs between runs, you first need to declare a variable in your script, which will then take the value from the maven parameters passed when running the script. However, in k6, you can have a hardcoded value for VUs in your script, run it, then run it again passing a flag and overwrite that value, without needing to touch the script.

Test Maintenance

Maintenance is easy using both as they’re designed with the “load tests as code” mindset and simulations should be stored in some sort of version control system. Gatling also supports a wide variety of protocols either natively or with help from its users’ communities while k6 only supports HTTP1.1 / 2 and WebSockets.

Reporting

When it comes to reporting, I’d say Gatling is the winner today since k6’s built-in reporting is not quite as advanced. One of Gatling’s best features is its automatic report generation, which creates a really nice webpage that allows you to clearly view and navigate the vast amounts of data logged by the tool. It’s even clearer when executing, while k6 displays little to no information on the terminal when running a simulation until it ends, Gatling has some real-time reporting. In any case, these tools can be integrated with some other open source solution to improve the real-time reporting, such as ELK or Grafana. This capability is important, for example, to cut a test run short if you see failures are happening. However, if integrating k6 with other tools for better data visualization is an option, you can produce really great reports that way.

Load Distribution

As for distributed load testing, it becomes a bit muddier. Both tools offer a paid, cloud-based version that better support reporting and distributed executions. Gatling however, also offers a free shell script which allows you to distribute your simulation amongst as many machines as you’d like. This script does require some previous configurations in all of the generators though.

On the other hand, the k6 team states that its tool is more performant than its competitors’, thus reducing the need for distributed load testing since you can supposedly generate serious load on a single generator, although we couldn’t check this claim (the test sites we were using started rejecting requests when the load got heavy). It seems to me though that this argument falls flat when, for example, you want to distribute the load between different geographic areas in order to simulate more real-world scenarios.  

The Final Verdict

If you’ve made it this far through my k6 review, thanks for reading—and I’ve got to say, I like k6, far more than I thought I would. I believe it has a lot of potential and it’s clear that the k6 team is highly involved, focused on improving it so that it’ll continue growing its user base and features.

I encourage anyone interested in new load testing tools to take a crack at k6 and let me know what you think about it in the comments! 

Recommended for You

3 Key Performance Testing Metrics Every Tester Should Know
14 Best Performance Testing Tools & APM Solutions