I just got the task, to advise on which performance testing framework to use for our product. As we already use Grafana, the suggestion coming from our Dev Lead, to examine whether it would be a good fit or not.
As we are using Karate DSL for API testing, a straightforward choice would be to go with Gatling, as it nicely integrates with Karate DSL, and we can reuse our existing tests for performance testing.
I compiled a list of important aspects to examine when choosing a performance and load testing framework. I reused the list from my Selenide vs. Selenium article and adapted it a little bit. I will use a simple scoring system but of course, again, you cannot simply decide based on the final score.
- Skillset available in the company
- Learning curve and ease of use
- License cost
- Integration with cloud testing services
- Test case reusability
- CI/CD integration
- Detailed reports
Skillset available in the company
Our team has experience with Karate DSL. The test cases written with Karate DSL using the Gherkin language can be reused with Gatling so Gatling seems to be a reasonable choice.
Grafana k6 requires JavaScript knowledge. Of course, our front-end developers are familiar with JavaScript, but our test automation engineers are not.
So the standing is Gatling:Grafana 1:0.
Learning curve and ease of use
In order to use any load testing framework, you have to be familiar with the theory of performance testing, which is needed independently from the tool. I am planning to take the ISTQB Performance Testing training and exam later this year.
Having the theoretical background, using Gatling is very simple, thanks to the reusability of the existing test cases.
As Grafana k6 requires JavaScript and learning a new syntax for testing code, it is one point again to Gatling. The standing is Gatling: Grafana 2:0.
License cost
Grafana k6
Grafana k6 has both a free and a paid version. Grafana k6 itself is an open-source framework that you can use free of charge.
The pricing of the cloud solution's Developer version is $99/month which allows 50 test runs monthly with a max duration of 15 mins, and it can only run on one thread.
The pricing of the cloud solution's Team subscription is $499/month and it allows 75 runs per month, which I still consider very little. The max run duration is 30 mins and you can run your tests on 2 parallel threads.
There is also a pro version for $1499/month which allows 100 test runs with a 60-minute time limit and 4 parallel threads.
There is also an Enterprise version which has a custom price depending on your needs.
Gatling
Similarly, Gatling has both a free and a paid version. Gatling is also an open-source framework. For the paid version you have the option to use the cloud service, or a self-hosted version.
The Scout cloud subscription costs 99€/month with a limit of 60 minutes test running duration and it has a 1-month data retention period.
The Scale version costs 396€/month with a 24hours maximum test execution duration and 3-months data retention period. This version also comes with a CI/CD plugin.
The Corporate version costs 1380€/month with more than 24 maximum test execution time and 6 months of data retention
The self-hosted version starts from 4800€/year.
However we won't use the cloud service in the short run, but we might use it in the long run. With the pricing of Gatling, it is a clear winner. Gatling: Grafana k6 3:0.
Integration with cloud testing services
This topic is the same as the one above, so the standing is still Gatling:Grafana k6 3:0.
Test case reusability
With Grafana k6 we have to start building our performance tests from scratch, and with Gatling, we can reuse our existing Karate DSL tests, so it is a clear win again for Gatling. Standing is Gatling:Grafana k6 4:0.
CI/CD integration
Grafana k6 can be integrated into your CI/D pipeline, you can find some instructions about it here.
Gatling free version can be also easily integrated into your CI/CD pipeline and from the cloud version, it is available from The Scale subscription.
This is a draw, so the standing is 5:1 for Gatling.
Detailed reports
Both Gatling and Grafana k6 have a detailed reporting with lots of useful charts.
Conclusion
It is a clear win for Gatling. So for our team, I suggest using Gatling framework for performance testing.
What is your number one performance testing framework? Please share your experiences in the comment section.
In case you do not want to miss my posts, just follow me here on Hashnode, on LinkedIn, on Twitter on Medium.com, on dev.to and even on Instagram. 😊
📚Join the Selenide community on LinkedIn! ✌