Automated tests — are they taking up too much resources & time to run?
“Automated tests are taking up too much resources and time to run!”
Does this sound familiar, especially when there’s continuous product development?
Adding new features and feature improvements are part of the product development process. With every new feature, there will be new automated tests. Feature improvements will lead to:
- Only modifying the existing test cases, or
- Creating new automated test cases on top of modifying the existing test cases
With new product features and feature improvements and in order to have good test coverage, the list of automated test cases will just keep growing; resource utilisations and the time to complete running the tests will be increased. Hence, it’s time to strategise the test execution process.
Oh yeah, there’s no one-size-fits-all test execution strategy!
The product’s type and requirements must be taken into consideration when planning for the test execution strategy. Of course, the test strategy will be improvised or changed along the way. We’ll be covering one of the simple test strategies, which will help to reduce the resources and execution time of the tests during time crunch.
Let’s take a simple scenario from a product (Task) and its three existing features:
- List task(s),
- Add a task, and
- Delete a task
Below shows the test cases of the three features in the mind map format:
| Note: This is not an extensive list for test scenarios and test cases.
When there are feature improvements or new features implemented, these test cases will (and must) also be executed. Now let’s take a look when a new feature, Edit Task, is added to the Task product.
As shown in the mind map above, the number of test cases increased quite a bit (if it’s not obvious to you, it’s the whole block on the left 😬). Since there’s an increase in the number of automated test cases, the resources and execution time will definitely increase too!
How can we reduce the resources and execution time when still achieving good test coverage?
We can start by breaking down the test cases into different test categories. When test categories have been mapped out, we can plan when and how often to execute the test cases. Let’s get started on the categorising of the test cases based on the Task product.
Test cases may and can be categorised into these types:
- Smoke Testing
Identify the major existing test cases in the product for testing. To ensure that all critical functionalities are working and there are no showstoppers, testing should be performed after every deployment to any environment. - Sanity Testing
Do an impact analysis. Identify the existing test cases that will/might be impacted by the newly developed feature or feature improvements. Testing should be performed before deployment to staging and production environments so as to ensure that the existing functionalities (especially those which may be impacted) still work as expected. - Regression Testing
Full set of existing test cases to be executed before deployment to staging and production environments to ensure that all existing features still work as expected. This can be scheduled to run once daily.
There are and will always have ongoing changes to the product as new features are being developed. With the test cases identified (a subset of the test cases) for smoke and sanity testing to be executed only after every deployment, resources and execution time will be reduced. Regression testing is also necessary to achieve good test coverage and we would try not to have any surprises that the existing features have broken, so we should try to schedule to do regression testing at least once daily.
Planning a test execution strategy isn’t easy — it requires a lot of time and effort to do so — and it will/may change over time. Thus, we hope this article has provided you some insights and help to kickstart your test execution strategy planning.
Remember to always start small and keep building up as you go!
🧙🏼♀Team Merlin 💛
Application security is not any individual’s problem but a shared responsibility.
