10 Suggestions for Choosing the Number of Concurrent Users for Your Performance Test
Do you know how many virtual users should you use for your website or app performance tests? If your answer is “No”, this is the right blog post for you to read. Even if you know Apache JMeter and Gatling, they can’t tell you how many concurrent users should be sending requests to the services you’re testing.
This is due to the fact that the number of virtual users to run a test is a business and product decision based on past user scenario patterns, future expectations, marketing initiatives, and product requirements.
So, if you’re a Test & QA Engineer, here’s a few suggestions to get you started for a performance test:
1. Check the Server Logs
Logs are essential for every software personnel. The server logs will contain the history of web requests, including the client’s IP address, date and time, and page requests. You may use this information to create a picture of how many real users visited your website in a given time and what activities they performed while surfing on it.
2. Compare Server Log Information With the APM Data
APM tools analyze user data in depth and helps with the identification of trends and scenario profiling. Check if anything is missing or inconsistent between the picture you got from the logs and the one you got from your APM tools, and you’ll have a better knowledge of how users acted on your website. Google Analytics is another option for these purposes. Also, you can calculate Ramp Up Time with G4 Google Analytics since it gives numbers of the all visitors and engaged visitor with their durations.
It’s also crucial to spot patterns in website growth. For example, if your traffic is consistently increasing by 10%, take this into account and increase the number of users in your test correspondingly.
3. Keep An Eye Out For Bottlenecks And High-Stress Points.
Investigate areas with the highest user activity, keep track of distinct trends, and see if your system was about to reach its limits at any time. Include those statistics in your testing because they represent possible crashing points in real time.
4. Check Product Requirements
Your product manager most likely determined on the capacity of your website or app when the test was planned, and as it is upgraded and improved. This choice was influenced by the company’s business strategy and aims. Verify the numbers with the product manager to ensure that they are included in the testing. Future plans are likely to be based on those numbers, so make sure that your app or website can handle the traffic. If they are unable to do so, plans should be made appropriately.
5. Check If There Are Any Upcoming Campaigns or Major Events
Campaigns, events, and holidays can all result in a surge in traffic and a corresponding boost in sales for your organization. However, if your website is down, this will not be possible. If something is scheduled, check with marketing and IT departments, and boost the number of users you are testing in preparation for these events.
6. A Thread Won’t Be Same With A Real Time User
Let’s think about an end-to-end scenario for an e-commerce website. A user logs in, goes to the main page, searches for a product, sees the results and goes to the detail page of one option. Then the user will continue with performing cart and payment steps. A real time user would be executing this scenario slowly. Because during the flow; he/she will think about his/her budget, try to find out the most charming option then fill the credit card info slowly. But a thread can execute this scenario much faster than a real time user, if you don’t add think time between requests in your script. So, 100 concurrent threads can provide bigger load compared to 100 real time concurrent users. Keep that difference in mind.
7. Start by Putting 80% Of The Capacity to The Test
Running your load test with the maximum capacity at the beginning will not be a good idea. The best practice is to improve the load gradually. Run the test with 80 percent of the capacity after you’ve decided on the numbers and keep an eye on your KPIs and how your system responds. Make sure that everything is entirely stable; memory capacity is low, CPU usage is low, and the system recovers quickly from surges. With 80 percent of maximum load, everything should be running smoothly. If anything feels unreliable, you can be sure that you won’t be able to rely on it at 100 percent.
8. Gradually Increase to 100% If The Test Was Successful
If the test with 80 percent capacity was successful, proceed to the 100 percent test. If this isn’t the case, look for bottlenecks, errors and make the necessary improvements and bug fixes. And then, it’s time to put your system under the test again. This is the predicted number of visitors for your website based on prior usage trends, trend research, product requirements, and anticipated events. Perform a memory leak check, as well as a CPU utilization check to see whether your server is acting unexpectedly or has made any errors. If something isn’t working right now, it’s unlikely that it will work in real time. Make the necessary improvements. Rerun the tests to ensure that your adjustments were applied correctly, and no performance degradation occurred.
9. Push Your System Beyond the Limits
Under the specified loads, your system performs admirably — excellent work! But be prepared for the unexpected! What if a competitor’s website goes down and you get a ton of traffic? What if a well-known figure makes a statement that generates a lot of interest to your product? Be prepared. The truth of the matter is that no system can be prepared for an endless number of users, but you can learn about your system’s flaws.
To do so, run stress and soak tests on your system and try to make it crash. After that, examine the data to determine where your weak points are. When you have this information, you can decide whether to make changes to your infrastructure or code. If something goes wrong in real time, you’ll know exactly where to investigate to fix the problem.
10. Review The Results of Earlier Tests
If you ran similar tests to your system before, reviewing these test results can provide you lots of useful information to choose several concurrent users. If your last test had faced 10.000 concurrent users at maximum capacity and couldn’t handle the load, your first goal may be reaching 10.000 concurrent users without any major issues. But if you were too far from 10.000 -let’s, say 6.000-, you can rerun the same tests to see if stats like total hit count, average throughput and response time have some progress.
That’s All:)
I hope these suggestions help you develop your performance testing scenarios.
