Or press ESC to close.

Making Sense of Performance Testing Results

Apr 25th 2023 12 min read
easy
performance
gatling3.9.3

Performance testing is a crucial part of ensuring that your application can handle the expected load and deliver a satisfactory user experience. But for many beginners, looking at the stress test report graphs can be overwhelming and confusing.

However, understanding these graphs is essential to make informed decisions about your application's performance and identifying potential bottlenecks before they become major issues. In this blog post, we will explore some of the most common stress test report graphs, such as response time distribution, requests per second, and response time percentile over time, and explain what they mean and how they can help you improve your application's performance.

We will use Gatling's stress testing reports as examples since it is one of the popular solutions for performance testing. However, it's worth noting that other tools will produce similar types of graphs.

Response Time Ranges

Let's start with an easy one. The Response Time Ranges graph in a Gatling stress test report displays the percentage of requests that fall within specific time ranges. This graph can help identify the distribution of response times and provide an understanding of the number of requests experiencing slow response times.

response time ranges graph

Suppose we're running a stress test on a web application that allows users to submit a form with various types of data, such as text, images, and videos. As part of the test, we send a series of requests to the application, such as submitting a form with text only, submitting a form with images only, and submitting a form with videos only.

After running the stress test, a report is generated that includes a Response Time Ranges graph. This graph displays the percentage of requests that fall within various time ranges, such as 0-500ms, 500-1000ms, and so on. We notice that the graph shows a high percentage of requests falling within the 0-500ms range, indicating that most requests are being processed quickly. However, there is also a significant number of requests falling within the 3000-4000ms range, indicating that some requests are experiencing slow response times.

By utilizing this information, we can delve deeper to pinpoint the specific requests that are causing the slow response times and take the necessary steps to optimize them. For instance, we may optimize the server-side code or employ a content delivery network (CDN) to enhance the delivery of images and videos.

Besides detecting performance issues, the Response Time Ranges graph can also aid in comparing the performance of distinct request types. For instance, we can compare the response times of requests with images to those without images to determine if the images are causing any performance issues.

Overall, the Response Time Ranges graph offers a valuable summary of the distribution of response times and can aid in identifying areas for optimization and improvement in the tested system.

Active Users along the Simulation

The 'Active Users Along the Simulation' graph in a Gatling stress test report shows the number of virtual users (simulated users) that are active at any given time during the test. This graph can help understand the load on the system being tested and identify any performance issues related to the number of active users.

active users along the simulation graph

Suppose we're running a stress test on a mobile banking application that allows users to perform various banking transactions, such as transferring funds and checking account balances. As part of the test, we simulate a large number of users accessing the application at the same time.

Once the stress test is complete, we produce a report that encompasses an 'Active Users Along the Simulation' graph. This graph illustrates the count of virtual users that are active at any given time throughout the test. We observe that the graph displays a sudden spike in the number of active users at a specific point during the test, which may indicate a performance issue linked to a particular transaction or feature in the application.

Using this information, we can investigate further to identify the specific transactions or features that are causing the spike in active users and take steps to optimize them. For example, we might optimize the server-side code or use caching to improve the application's performance.

In addition to identifying performance issues, the 'Active Users Along the Simulation' graph can also help understand the capacity of the system being tested. By analyzing the graph, we can determine the maximum number of active users that the system can handle before experiencing significant performance issues or errors.

Response Time Distribution

The 'Response Time Distribution' graph shows the distribution of response times for the requests made during the test. This graph can help identify the overall performance of the system being tested and any bottlenecks or performance issues that may exist.

response time distribution graph

Imagine we're running a stress test on an e-commerce website that allows users to browse and purchase products. As part of the test, we simulate a large number of users browsing different categories of products and adding items to their carts.

After executing the stress test, a report that contains a 'Response Time Distribution' graph is generated. This graph illustrates the response time distribution for the requests executed during the test, with the X-axis representing the response time in milliseconds and the Y-axis showing the percentage of requests with that response time. It is apparent from the graph that there is a high percentage of requests with response times below 500ms, indicating that most requests are processed rapidly. However, there are also a considerable number of requests with response times exceeding 2000ms, suggesting that some requests are experiencing slow response times.

Using this information, we can investigate further to identify the specific requests that are causing the slow response times and take steps to optimize them. For example, we might optimize the database queries or use caching to improve the performance of the application.

Besides detecting performance problems, the 'Response Time Distribution' graph can also facilitate a comparison of the performance of various sections of the system. For instance, we could compare the response times of requests related to product browsing to those related to checkout to determine if one section of the system is causing any performance issues.

Response Time Percentiles over Time

The 'Response Time Percentiles over Time' graph shows the distribution of response times for the requests made during the test over time. This graph can help understand how the performance of the system being tested changes under different levels of load.

response time percentiles graph

Suppose we're running a stress test on an online gaming platform that allows users to play games with each other. As part of the test, we simulate a large number of users playing games and interacting with the platform at the same time.

Upon running the stress test, we create a report that features a 'Response Time Percentiles over Time' graph. This graph demonstrates the response time percentiles (such as the 50th, 75th, 90th, and 99th) for the requests that were made throughout the test over a period of time. We observe a gradual rise in response times depicted in the graph as the number of users increases. Furthermore, at specific intervals during the test, we note a significant increase in the 99th percentile response time.

Using this information, we can identify the specific parts of the system that are causing the slow response times and take steps to optimize them.

In addition to identifying performance issues, the 'Response Time Percentiles over Time' graph can also help understand the capacity of the system being tested. By analyzing the graph, we can determine the maximum number of users that the system can handle before experiencing significant performance issues or errors.

Number of requests per second

The graph, as implied by its name, displays the number of requests that are made to the tested system over a period of time. This visualization can be valuable in comprehending the system's ability to manage various levels of load and detecting any potential performance difficulties that may emerge.

requests per second graph

Assume we're running a stress test on a social media platform that allows users to post and interact with each other's posts. As part of the test, we simulate a large number of users posting and interacting with posts at the same time.

After running the stress test, we generate a report that includes a 'Number of Requests per Second' graph. The graph shows the number of requests being made to the system over time, with the X-axis showing the time and the Y-axis showing the number of requests per second. We notice that the graph shows a gradual increase in the number of requests being made as the number of users increases, with occasional spikes in the number of requests during certain periods of the test.

Using this information, we can identify the specific periods of the test when the system is experiencing high levels of load and check the performance of the system during those periods. For example, we might investigate whether the response times are increasing during these periods or if the system can handle the load without any significant performance issues.

In addition to identifying performance issues, the 'Number of Requests per Second' graph can also help identify the maximum number of requests that the system can handle before experiencing significant performance issues or errors.

Number of responses per second

As you might have guessed, the graph shows the number of responses received from the system being tested over time. This graph can help understand how the system responds to different levels of load and identifying any performance issues that may arise.

responses per second graph

Imagine we're running a stress test on an e-commerce platform that allows users to browse and purchase products again. As part of the test, we simulate a large number of users browsing and purchasing products at the same time.

After running the stress test, we generate our report that includes a 'Number of Responses per Second' graph. The graph shows the number of responses received from the system over time, with the X-axis showing the time and the Y-axis showing the number of responses per second. We notice that the graph shows a gradual increase in the number of responses received as the number of users increases, with occasional spikes in the number of responses during certain periods of the test.

Using this information, we can identify the specific periods of the test when the system is experiencing high levels of load and check the performance of the system during those periods. For example, we might investigate whether the response times are increasing during these periods or if the system can handle the load without any significant performance issues.

Apart from detecting performance concerns, the 'Number of Responses per Second' graph can also aid in determining the maximum amount of responses that the system can manage before encountering substantial performance problems or errors.

The 'Number of Requests per Second' and 'Number of Responses per Second' graphs are related in that they both show how the system being tested is handling incoming requests and generating responses. However, they represent slightly different aspects of the system's performance under load.

When the 'Number of Requests per Second' graph depicts a significant amount of requests sent to the system, but the 'Number of Responses per Second' graph illustrates a lower quantity of responses received, it could signify that the system is encountering performance issues or errors that hinder it from responding to every incoming request.

On the other hand, if the 'Number of Requests per Second' graph and the 'Number of Responses per Second' graph both show a high level of activity, it may indicate that the system is handling the load well and can generate responses quickly.

Response time values

In a stress test report, the 50th, 75th, 95th, and 99th percentile values in response times represent the time it takes for a given percentage of requests to be completed by the system being tested.

For example, the 50th percentile, also known as the median, represents the response time for the request that falls right in the middle when all requests are sorted in ascending order based on their response times. In other words, half of the requests were completed faster than this value, and half were completed slower.

The percentile values are useful because they provide a way to understand the distribution of response times across all requests. By looking at the percentile values, we can see how long it takes for the majority of requests to be completed, as well as how long it takes for the slowest requests to be completed.

In addition, percentile values can help identify outliers or requests that take significantly longer to complete than others. By focusing on the 95th or 99th percentile values, we can identify the requests that are taking the longest to complete and investigate potential performance issues or bottlenecks in the system.

response time percentile values

Besides the percentile values, we also have the min, max, mean, and standard deviation.

The minimum response time (min) is the shortest amount of time it took for a request to be completed during the test. This value can be helpful to identify the best-case scenario performance of the system.

The maximum response time (max) is the longest amount of time it took for a request to be completed during the test. This value can be helpful to identify potential bottlenecks or issues in the system.

The mean represents the average response time across all requests. It is calculated by adding up all the response times and dividing by the total number of requests. The mean is useful because it provides an overall picture of the system's performance, but it can be skewed by outliers or requests that take significantly longer or shorter than the rest.

And the standard deviation measures the spread of the response time values around the mean. It provides a way to understand how much the response times vary from the average. A smaller standard deviation indicates that the response times are tightly clustered around the mean, while a larger standard deviation indicates that the response times are more spread out.

The mean and standard deviation provide a way to understand the central tendency and variability of response times across all requests. By looking at the mean and standard deviation, we can get a sense of how long requests are taking on average, as well as how much they vary from that average.

Nonetheless, it is vital to bear in mind that the mean and standard deviation may not provide a comprehensive account of the system's performance. These metrics can be influenced by outliers or other variables that may not accurately represent the majority of requests. As a result, it is essential to examine other metrics, such as percentile values and graphs, to obtain a more comprehensive understanding of the system's performance when under load.

Let's recap:

Stress testing generates a report that contains various metrics and graphs that help comprehend a system's performance under load.



These reports typically include graphs that depict the number of requests and responses over a period of time, providing insight into how the system handles different levels of load and identifying potential performance issues.



Other metrics, such as percentile values, can be used to determine the response times for various requests and the system's ability to handle different levels of load.



While mean and standard deviation can provide useful information, they may not always tell the whole story and can be affected by outliers or other factors. Therefore, it's important to consider other metrics as well.



By analyzing stress testing reports, businesses and developers can gain a better understanding of how their systems perform under stress and identify areas for improvement to ensure better performance and reliability in the future.

Stay tuned for a more advanced blog on stress testing, where we will explore more advanced techniques and best practices for stress testing. Until next time.