By Ariadna Trueba, Mario Losada Rincón and Santiago Agudelo, Parser
When embarking on the creation of a new automation framework, several critical aspects come into play. Among the foundational pillars underpinning its success is the architecture. The efficacy of a test automation framework hinges on the intricacies of its architecture and how its implementation influences performance. Let’s delve into the key considerations when building a test automation architecture.
— –
When discussing test automation, people often prioritise quantity over quality. This implies creating more test cases regardless of the quality of the test automation scripts. While the test coverage is one of the critical testing factors, we need to think about quality first. After ensuring quality, we can focus on increasing the number of test cases to attain the desired coverage.
In the realm of test automation, quality starts with the architecture. This is the key part that would decide whether a test automation project is successful or not. A well designed architecture can lead us to achieve better results, more reliability, and confidence with the usage within quality assurance along with the test coverage. The architecture should define a high level of abstraction by defining reusable components and modules that can be used in multiple test cases. This method can enhance the maintainability, flexibility, and extensibility of the test automation framework. Overall, abstraction is a powerful technique that can help us to simplify complex systems and improve the maintainability and flexibility of the test automation architecture. By abstracting test cases, data, system, and framework components, we can create a modular and reusable test automation architecture that is easier to modify, extend, and maintain. Therefore, our test automation architecture would support quality attributes like the following ones:
Maintainability: This is one of the greatest concerns in test automation projects. Sometimes, there is great automation with great performance and coverage, but when a change is needed, the cost is higher than implementing a new feature, which leads the automation project to fail. Maintenance of the automation tests should not consume a significant part of the test effort, ensuring the tests remain effective in the long term.
Performance: Even if the automation is intended to be executed without human intervention, execution time is an important consideration to take into account. We should use as many machine resources as we can and leave aside other threads that can affect the performance. There are three common mechanisms to consider in order to assure that behaviour:
- Hard-coded waits: This is not recommended as it is directly a waste of the machine’s resources. This can be the root cause of many test automation problems and instability.
- Dynamic wait by polling: This mechanism is acceptable but it is still a waste of resources as we are dynamically waiting for some state or action to happen and we could use these resources to run other tests.
- Subscribe to the event mechanism of the SUT (System Under Test): This is the most recommended and flexible way as it is much more reliable than the two previous options. However, the test scripting language needs to be subscribed to the SUT event and it needs to offer these events to the test application.
In any case, the most used mechanism nowadays is the dynamic wait.
Learnability: The use of consistent and standard terminology throughout the whole project can help users understand quickly and easily along with concise, clear, and up-to-date documentation that helps the test automation engineers to know how the system is structured, how to modify it, and how to troubleshoot issues. Additionally, designing a simple system while still fulfilling the requirements is the key to learnability.
Modularity: A Test Automation Architecture should be used with as much abstraction as possible, since this enables flexibility regarding further evolution or transition to new approaches or technologies. A good modular design fitting the SUT allows a high degree of reusability of individual components and thus reduces the number of artefacts that need to be adapted when changes are necessary, which also improves maintainability.
Reusability: A reusable system can save time and effort in developing new test cases and maintaining the existing ones. A modular architecture is not only beneficial for maintainability but also for reusability. The main points are:
- Including the abstraction at the architecture level to take it to the framework level
- Having test case libraries for easy reuse
- Defining parameters for test cases so they can be easily modified and reused, which leads us to reduce the amount of duplicated code and make it easier to create and maintain the test cases.
Reliability: Automation tests should be designed to be reliable, which means, they should be able to run consistently and the duration should be similar between each execution. This can be achieved through robust error handling, fault tolerance, and the use of reliable tools.
Scalability: The test automation architecture should be designed to be scalable. This means the tests can be easily modified in size or scale. This can be achieved through a flexible architecture, the use of reusable components, and the ability to handle large amounts of data.
Interoperability: The architecture should be able to work with other tools and systems. This can be achieved through the use of open standards and APIs, as well as effective integration with other operating systems.
Principles for the Test Automation Architecture
Test automation architecture does not differ from the development architecture, as both are software products. Therefore, we need to use best practices and principles to build our automation architecture. One of these known principles is called SOLID and it can help us to achieve the needed quality attributes.
SOLID is a set of five design principles widely used in object-oriented programming. These principles aim to make software systems more maintainable, flexible, and extensible. Here’s how SOLID principles can be applied in automation:
Single responsibility (SRP): This principle states that a class should have only one responsibility. In automation, this means that each test case should have a single responsibility or goal. This makes the test case easier to understand, modify, and maintain. The responsibility must be completely encapsulated in this component, like for example, by generating keywords, creating test scenarios, logging results, etc.
Open/Closed Principle (OCP): This principle states that a class should be open for extension but closed for modification. In automation, this means that the test automation framework should be designed to allow new features to be added without changing the existing code. This makes it easier to extend the framework and add new test cases. The main principle here shall be called “Closed for modification = do not break backward compatibility”.
Liskov Substitution Principle (LSP): This principle states that objects of a superclass should be able to be replaced with objects of a subclass without affecting the correctness of the program. In automation, this means that test cases should be designed to work with different types of inputs or data sources. This makes it easier to reuse test cases and run them with different data sets.
- Components must be replaceable without affecting the overall behaviour of the system
- Also called behavioural subtyping components and their replacement must expose the same observable behaviour
- E.g. When ordering a taxi, it should not matter which car brand the taxi has. A Renault, taxi, or a Toyota taxi, can “substitute” a taxi
Interface Segregation Principle (ISP): This principle states that a class should not be forced to depend on interfaces that it does not use. In automation, this means that test cases should only depend on the interfaces or APIs they need to interact with. This reduces dependencies and makes it easier to modify or replace the system being tested.
- No component should be forced to depend on methods that it does not use
- Multi-purpose components
- E.g. To get your car out of the garage, you need a key to the garage. You should not need a key to your cellar.
Dependency Inversion Principle (DIP): This principle states that high-level modules should not depend on low-level modules, but both should depend on abstractions. In automation, this means that the test automation framework should be designed to depend on abstractions rather than concrete implementations. This makes it easier to modify or replace the system being tested and improves maintainability.
- Decouple components
- High-level modules shall be independent of low-level implementation details
- Components should not depend on specific automation scenarios
- E.g. When using the mouse, it should make no difference whether it is connected via serial port, USB, or Bluetooth
Additionally, it is recommended to use other principles to enhance code quality and embrace best practices, such as the KISS principle and the DRY principle.
- KISS Principle (Keep It Simple, Stupid!): this principle is based on keeping things simple and avoiding unnecessary complexity in software design and development. It aims to make solutions easy to understand, maintain, and extend, avoiding excesses or complications that could hinder the development process.
- DRY principle (Don’t Repeat Yourself): This principle promotes code reusability and the elimination of duplication in a software system. It’s about avoiding repeating logic or functionality in places, which facilitates maintenance and improves code consistency.
In software automation, applying these principles means creating test scripts or frameworks that are simple, concise, and easy to understand, avoiding unnecessary repetitions and maintaining modularity to reuse components in different tests. This improves development efficiency and facilitates the detection and correction of errors.
Finally, it is also important to select the most adequate design pattern for test automation, typically it is recommended to use the structural Page Object Model (POM), but there are some cases where the project nature implies a code growth that could break the SOLID principle. To avoid this situation and confront a possible refactor, proper planning must be done and if this is the case, consider the behavioural Screenplay design pattern.
The main benefit of the POM is that it separates test logic from the page structure, making it easier to make changes to the application’s design or behaviour without significantly affecting the test scripts. It also promotes the reuse of methods in different tests and increases readability and maintainability of the code.
- Page Object Model (POM): This design pattern is a technique used in software test automation to improve the maintainability, reusability, and clarity of test scripts. Here, each page or screen of a web or mobile application is represented as a separate class. These classes contain the specific elements and behaviours of the page. Then, methods are created to interact with those elements and perform actions
Screenplay, here test scripts are written as stories that describe the interaction between actors and the tasks they perform. This makes scripts more readable and easier to understand for non-technical people. Additionally, task reuse is simpler, as they can be combined and composed flexibly to form complex test flows.
- Screenplay: This design pattern is another technique used in software test automation to improve the clarity and maintainability of test scripts. Instead of focusing on individual pages, as in POM, Screenplay focuses on actors and the tasks they perform. Actors represent the users or systems involved in the test, and tasks are specific actions that these actors can perform.
In summary, both design patterns are used in test automation to improve organisation, reusability, and clarity of scripts, regarding their different design approaches.
The ones described above are the main considerations to take into account when designing and implementing the test automation architecture. Achieving these objectives can make the difference between success or failure in our test automation project, as well as helping to improve quality and efficiency during the same process.
The benefits of a test automation framework
A test automation framework is a set of guidelines, rules, and tools that are used to automate the testing process. It provides a structured approach for writing, executing, and maintaining test cases, but the Test Automation Framework can be used to realise a Test Automation Solution, which in turn is the realisation of a Test Automation Architecture (see image 1).
Image 1. TAA, TAS + TAF
Taking into account all the described considerations in the design and implementation of the Test Automation Architecture, the following benefits can be easily achieved:
Easy to use: A good test automation framework should be easy to use, even for testers who are not experienced in automation. This can be achieved through intuitive interfaces, clear documentation, and easy-to-understand workflows. When the framework is easy to use, testers can focus on creating and executing test cases rather than struggling with tools.
Well-documented: Clear and comprehensive documentation is a critical component of any test automation framework. Documentation shall include information on how to set up the framework, how to use it to create and execute test cases, and how to troubleshoot common issues. When the framework is well documented, testers can quickly and easily find the information they need to get their work done.
Maintainable: A test automation framework that is designed with maintainability in mind is much easier to update and modify as needed. This can include modular design, well-organised code, and good commenting practices. When the framework is maintainable, it is easier to add new test cases, update existing ones, and keep the framework up-to-date with changes in the application being tested.
Consistent test automation approach: A test automation framework provides a consistent approach to creating and executing test cases. This helps ensure that all testers are following the same process and using the same tools, which leads to more consistent and reliable results. When the framework provides a consistent approach, it is easier to compare results across different test cases and identify trends or patterns.
Overall, a well-designed test automation architecture should offer a scalable, reusable, maintainable, and consistent approach to test automation. It needs to be flexible enough to accommodate changes as needed while providing robust and accurate results promptly. A good test automation architecture shall significantly improve the efficiency, accuracy, and reliability of software testing efforts.
The architecture should be designed to handle a wide range of testing scenarios, including different environments, data sets, and user interactions. This shall also be flexible enough to accommodate changes in the application being tested, as well as changes in the testing requirements or tools being used.
Furthermore, a solid test automation architecture shall maintain a consistent approach to test automation, ensuring that all quality teams follow the same process and use the same tools.This promotes more reliable and consistent results, which helps to identify trends or patterns. Additionally, the architecture should be easily maintainable and updatable, allowing the testers to save time and effort while maintaining consistency and accuracy. This ease of maintenance facilitates the creation of reusable test cases and scripts, while also supporting large-scale testing endeavours. A delay of one second or a bad practice must not be allowed for an easier implementation because the slightest change may cause a global affectation. It is essential to pay attention to all details to have the desired quality.
