REST Assured vs. JMeter: A Comparison of REST Test Tools

Automated testing is a critical component of the software quality assurance process. But testing representational state transfer (REST) APIs can be a tricky business. Many who resort to manually testing REST APIs overlook details and miss out on ways they could save time.

Any REST API service being tested properly will have these areas covered:

• Testing of whether the API service complies with REST standards
• Testing of API service response codes
• Assertion of API service response content
• Performance benchmarking of API service

For this, you need the right tools. Apache JMeter and REST Assured are two popular REST test tools for testing REST API services. You can learn more about how you can use them in practice from here and here.

In this article, you will learn the pros and cons of both of these tools and where they fit best in your existing REST API projects.

Ease of Setting Up

There is one prerequisite: Both JMeter and REST Assured require Java, so you need to have Java installed on your computer in order to run JMeter or REST Assured.

Installing Apache JMeter is very simple: Go to the download page, pick the ZIP or TAR, download the archive, unpack it, and you’re ready to go.

Go to folder where you have unpacked the archive, go to bin subfolder, and run jmeter.bat (or similar for your operating system, if other than Windows). For more detailed information on how to set up Apache JMeter, see this page.

Installing REST Assured is a bit of a different experience. REST Assured is not a standalone tool. It is a Java library. In order to use it, you have to set up a new Java project and include it as a library for your project. Of course, there is support for many build tools, so you can just point your dependency resolver to some public (Maven) repo. You can find more detailed information on how to set up REST Assured here.

Documentation

Apache JMeter documentation can be found on the official JMeter website. It provides some pretty thorough documentation from getting started to complex features such as listeners. Also, in the official JMeter website, you can find links to some very rich resources such as video tutorials. If you are less into reading and more into watching, this is definitely a plus for you.

REST Assured documentation, as the tool/library itself, can be found inside the Github repository. It has a big list of features listed in online documentation with very detailed explanations of how to achieve something. However, the entire documentation is one huge web page—not the most user-friendly way of exploring a tool.

Automated REST Test Creation and Maintenance

Apache JMeter, as a standalone application with a graphical user interface (GUI), allows you to create your tests from the GUI. This can be handy especially because of the entities hierarchy introduced from JMeter—everything you create inside your test will be shown in a tree-like representation in JMeter GUI.

In order to create a test, one does not have to have any coding skills. However, the GUI does not limit any of the possibilities regarding test planning. If you would still like to add some custom programming/flow, BeanShell component(s) will allow you to do that.

When you save your JMeter project, all that you have created will be saved to one huge XML file (with the .jmx extension). Generally, even for a small change in one of tests, you will (very likely) have to reopen the project in the GUI in order to make the change.

If you need to collaborate with others on the same JMeter project file, this huge XML file will not be easy to maintain using any of source control management (SCM) systems (such as Git, SVN, Perforce, etc).

REST Assured, on the other hand, requires Java coding skills to create the tests. There is no user-friendly GUI for test creation. However, REST Assured allows you to write the tests using Gherkin language (Given-When-Then syntax) so even if you need some Java coding skills, tests are fairly human readable.

given(http://via.placeholder.com/2000x200?text=image+placeholder).
    config(RestAssured.config(http://via.placeholder.com/2000x200?text=image+placeholder).jsonConfig(jsonConfig(http://via.placeholder.com/2000x200?text=image+placeholder).numberReturnType(BIG_DECIMAL))).
when(http://via.placeholder.com/2000x200?text=image+placeholder).
    get(“/price”).
then(http://via.placeholder.com/2000x200?text=image+placeholder).
    body(“price”, is(new BigDecimal(12.12)));

Since REST Assured is a library and tests are actually written in Java, you can organize your tests in separate Java files so when you need to change one test, you just need to open one file. You don’t need a GUI, so you can edit the test easily even through an SSH terminal. Collaboration with others is fairly simple because it is easy to maintain Java files using any SCM.

Types of REST Tests

JMeter is well known as a performance test tool for testing web apps and services. Out of the box, it has nice reporting options with different chart types that you can use to show the performance test metrics.

On the other hand, also it is not the primary usage scenario for JMeter. You can create functional tests for REST APIs with JMeter, including response assertion.

REST Assured’s primary usage scenario is for functional testing of REST API services. It even comes with its own response assertion mechanism (the Gherkin syntax). Since REST Assured is a Java library, it can easily be integrated with unit tests.

When it comes to performance testing, REST Assured does not support performance testing out of the box. If you, however, would like to use it for performance testing, you would have to develop your own custom framework for performance testing around REST Assured.

Test Execution

JMeter tests can be invoked in two ways.

From the user interface by clicking on Run button:

From the command line:

jmeter -n -t your_script.jmx

The official recommendation from JMeter is to not run the performance tests using the GUI, as the tests will run slower (in terms of number of requests per second) compared to running them from command line.

REST Assured, on the other hand, requires a separate test runner, for example jUnit or TestNG (or any other way for executing Java code).

From Eclipse using TestNG:

From the command line using, for example, the Ant build tool:

ant run-tests

Types of Reports

Apache JMeter, due to its primary purpose, has a large number of report types that you can create from the results of your tests. Since you can run performance tests from JMeter, performance metrics are usually shown using charts like this:

You can learn more about JMeter reports and the types of reports you can create from here.

REST Assured itself does not create any reports, so it is up to you and a test framework you chose. However, when integrated with some unit testing framework, you can get pretty usable reports in HTML, text or XML formats.

Integration with Jenkins

When it comes to continuous integration of products under test, it is very important to integrate your test automation framework with CI tools for both running tests and reporting on results.

JMeter automated tests can be easily integrated with Jenkins using the Performance plugin which allows you to set up Post-build actions and to get a nice-looking dashboard page out of a Jenkins job page.

REST Assured can also be easily integrated with Jenkins because Jenkins has plugins for almost all testing frameworks; for example, the TestNG plugin which again can create a solid trend chart for test results.

Community Activity

This a category that cannot be measured easily. Community activity depends on a number of factors such as maturity of the tool or number of contributors.

Apache JMeter has a larger community nowadays. This is mostly because of the fact that JMeter showed up in the market long ago and is considered as industry standard when it comes to performance testing of web apps and services. JMeter publishes official releases in every 5-6 months. Its GitHub repository is very active, with new commits on a daily basis. JMeter uses Bugzilla as issue tracker and has a list of open issues that are frequently changed. There are 23000+ results for JMeter in StackOverflow search. From the social networking sites, JMeter has an official Twitter account. On Facebook and LinkedIn, there is no official account for JMeter, but there are couple of groups, the larger of which is this Facebook group and this LinkedIn group.

REST Assured is cutting its way to the main stage and has become the first choice, a recommendation from multiple sources, when it comes to automating REST API service functional tests. REST Assured has been publishing official releases once every 6 months for the last two years. The GitHub repository has a lots of contributors and is a bit less active than JMeter’s, but it is also very active. REST Assured is tracking issues in the GitHub repository, so all is in one place. In StackOverflow, there are 2000+ search results for REST Assured, so we can say it’s significantly lower compared to JMeter search results. As for the social networking groups, only the LinkedIn group was found with significantly lower number of members compared to JMeter. The REST Assured community may not be as big as the JMeter community, but it is very active.

Summary

Both JMeter and REST Assured have their advantages over each other. But based on the project that you are working on, one will be a better fit than the other for your REST API test automation. Both are very robust, have useful testing options, boast nice reporting capabilities, and can easily be integrated with custom testing frameworks.

Here is a summary of all of the comparison criteria explored in this article for these two popular REST test tools: