Micro tests, macro impacts: the benefits of switching to Atomic testing
Small tests can be surprisingly powerful in test automation, especially when compared to traditionally lengthier end-to-end tests. End-to-end testing, while invaluable for real-world test coverage, often has significant downsides. These tests tend to be slow, prone to errors, and often create scaling bottlenecks. To improve test performance, reliability, and scalability, we have to move beyond the constraints of end-to-end test automation.
This is where a shift towards smaller, more targeted tests becomes crucial. Atomic tests, which are defined by their focused approach and isolated scope, challenge the constraints of end-to-end tests whilst retaining their benefits. They are a powerful response to end-to-end tests, but before we delve into Atomic tests further, let’s take a closer look at the problems Atomic tests help solve.
The problems with end-to-end tests
End-to-end tests are useful for simulating real user scenarios, primarily through the UI, to validate the complete flow of an application from start to finish. Testing an application through the UI is invaluable for ensuring all application components and dependencies are working in sync. But automating end-to-end tests is also precisely problematic because of their over-reliance on the UI for testing.
Each additional UI interaction in an end-to-end test increases the test complexity and the likelihood of generating a test error. These errors can stem from various factors, including dynamic page content, timing and synchronisation issues, and unpredictable internal or external dependencies. Put simply, the longer an end-to-end test, the more likely it will break.
The likelihood of test errors due to the UI is also compounded by other common issues in end-to-end tests. We’ll outline three key problems that Atomic tests can solve below, with the help of an example test that adds an item to a cart in an online store application:
- Lengthy execution time: End-to-end tests often take a long time to run due to their comprehensive coverage of the entire application flow. In the example below, each setup step in the workflow (blue boxes) needs to be executed to perform the key actions and validations (yellow boxes) for the AddItemToCart test i.e. the key test steps are adding to a cart and then validating the cart, but we cannot add an item until we’ve logged in, then searched and selected a product. This extensive workflow prolongs the total execution time of the test.
- Sensitive to changes: End-to-end workflows can be highly sensitive to changes in more than one area of the application, often triggering a chain of failing steps. For example, a test failure in the ‘Select Product’ step below would prevent the key steps (in yellow) from executing. This makes the test needlessly brittle and prone to signalling false negative results.
- Lack of test isolation: Often many end-to-end tests share common setup steps which can trigger multiple test failures. For example, a test failure in the shared ‘Select Product’ step below would not only cause AddItemToCart to fail, but it will also fail unrelated tests as shown. This produces excessive noise, instead of accurately identifying the source of failures.
Summing up, end-to-end tests have a wider surface area for UI interaction, resulting in tests that are slower to execute, more prone to failure, and often with inaccurate results. Though invaluable for confirming real-world functionality, these tests evidently require a more considered balance between coverage and reliability. Let’s unpack how Atomic tests help us achieve that.
Shifting to Atomic testing
Atomic tests are focused tests that cover a very specific software function. The term Atomic here implies the smallest sensible unit of test possible to validate a piece of functionality. Atomic tests are somewhat contrary to end-to-end tests because Atomic tests are designed to be functionality-focused, minimal, and independent.
These three points can be regarded as the guiding principles for designing Atomic tests, but the specific techniques to implement them can vary; including the use of session cookies, browser javascript functions, application API calls, and backend/database operations. Let’s expand on the three key terms a little further with the previous AddItemToCart example:
- Focused functionality: Each test is intended to be isolated and limited in scope to assess only the functionality under test. For example, The AddItemToCart test should be limited to the key actions and validations only, while abstracting away any other steps as much as possible. This should make test execution more efficient.
- Minimal setup and teardown: Atomic tests use minimised setups and teardowns, particularly by reducing UI interactions to focused areas as much as possible. For instance, we can reduce the UI interactions for AddItemToCart to key steps (in yellow) by abstracting away setup and teardown components with methods and operations that aren’t dependent on the UI (depicted by the grey boxes). The goal is to meaningfully isolate the UI test only to what is necessary and nothing more, resulting in more reliable tests.
- Test isolation: Atomic tests are intended to be run independently of each other. This ensures that the failure of one test does not impact the results of another. For example, if we were to split our earlier end-to-end test AddItemToCart, we’d have (at least) two independent tests with minimal external impacts. This also helps us identify the source of an error more efficiently, making test results more accurate.
We can achieve greater test atomicity by abstracting away setup and teardown components with methods that do not depend on the UI. And by reducing the UI automation surface area, we can achieve faster execution, greater reliability, and higher accuracy. These gains also set the foundation for expanding our test coverage in a more scalable manner. Now that we understand the mechanisms and benefits of Atomic testing, let’s quickly tackle the question of how much atomicity is needed for a test to be considered Atomic.
How atomic is atomic?
Traditional end-to-end testing strategies often link comprehensive coverage with extensive, multi-step tests. Atomic testing disrupts this norm by dissecting complex feature sets into smaller units. Each subsequent unit is converted to individual standalone tests.
The process of breaking down extensive tests into concentrated units offers an additional advantage: it compels us to thoughtfully consider each functional component of an application before isolating it. This exercise requires a delicate balance — achieving the minimal scope necessary while ensuring that no critical aspect, such as testing key dependencies, is missed. This is in line with my earlier description of Atomicity as the smallest sensible unit of a test designed to verify a specific piece of functionality.
The definition of what constitutes the ‘smallest sensible unit’ could be debatable. Nicolay Advolodkin suggests that Atomic tests should ideally have just two, or at most three, assertions. Simplifying your tests to this degree might sound challenging, but even small steps towards this goal could enrich your tests with the core characteristics of atomicity: focused functionality, minimalism, and independence. Adopting these can significantly boost your test suite’s overall effectiveness.
Final thoughts
Shifting towards atomicity can have a powerful impact on testing. By embracing the principles of focused functionality, minimal setup, and independence, we’ve seen how Atomic tests can potentially transform not just individual tests, but entire automation test suites.
The concept of Atomic testing is not just a theoretical ideal but a powerful practical approach that delivers tangible benefits. Faster execution times, improved accuracy, and simpler maintenance are not just aspirations but achievable realities with Atomic tests. Whether it’s a small feature or a complex system, the principles of atomicity are a powerful guide for crafting more efficient, effective, and reliable testing practices.
Like this article?
Follow me or find me on Twitter and LinkedIn.
References
I found the following resources invaluable for my understanding of atomic testing. If you’d like to learn more about Atomic tests, be sure to check them out:
- An excellent resource with code examples by Nikolay Advolodkin: Automated Atomic Tests (Definitive Guide)
- This excellent podcast episode by the inimitable Joe Colantonio with Nikolay Advolodkin: Why Automated Tests Should Be Atomic (The Atomic Punk)
