Welcome to this second part article. I assume that at this point, you can be starting test automation from scratch, hitting a critical wall, or feeling some warning signs of Test Debt.
We are now moving on to building test automation wealth in the first place and maintaining it. This is a true challenge in dynamic ecosystems and organizations when test automation value is not necessarily shared. This is where Quality Engineering constraining the system to quality enters the game.
This article shares the key elements of Quality Engineering that will be soon released in the ebook (more info at the end of the article). It covers how to apply the architecture, methodology, organization, skills, and management for test automation, highlighting the key benefits.
These practices come from our round-table “Forget Test Debt: Build Test Automation Wealth”. I thanks each of the participants for their participation and contribution:
- Joel Oliveira, Software Testing & Quality Assurance Evangelist, Public Speaker, Trainer, Senior Program Manager, Independent Consultant
- Rafael Amaral, Senior Software Engineer in Test at Farfetch
- Filipa Nogueira, Engineering Team Lead at La Redoute
Let’s start by defining wealth in test automation.
Follow the QE Unit for more exclusive Quality Engineering from the community.
What is Valuable Test Automation?
The test automation wealth is about creating value. It is the definition of quality to be value to someone who matters, as stated by Michael Bolton. We can precise it in the context of software engineering.
The value of test automation is not in the tests themselves; it is about the impacts on overall goals within the organization. Therefore, we must take a step back from the sole perspective of testing: what can we bring to the overall engineering perimeter? Most importantly, what can we bring to the Business? It is important to look broadly. We can deliver Quality without Testing. Quality is also about maximizing results with the least effort, hence the necessity to remove waste.
The wealth of test automation is defined by the stakeholders: your users, your business, and IT teams. Test automation is usually valued for improving the user experience, even in production leveraging SRE and Observability. It can also mean a faster value proposition and creation to your users by enabling faster cycles times of your software changes. The Accelerate metrics are supporting indicators you can use as a reference. Questions are powerful to understand better the ranking of criteria: Are we trying to reduce the risk? Gain trust in our deliveries? Increase reliability?
The wealth to build is therefore contextual, building up upon your stakeholders and leveraging benchmarks. The shared understanding you can create is already wealth by streamlining the activities on the right priorities. One common mistake of test automation wealth is to make the wrong parallel with money. For automated tests, we don’t always need more of them. Less is more (that’s also a debate for money, not in the scope of this article).
Defining the value is a first step. The next one is to translate it into reality.
Architecturing the Test Automation Wealth
We know the shared goals and priorities from the previous exercises. However, we don’t necessarily know how to translate the requirements inside our test automation plan effectively. This is the goal of architecture, transforming ideas into reality.
Our first action is to define what, to then prioritize its scope. The What is usually about which use-cases are most valuable declining the adequate test techniques. A particular scope can be even more valuable when performed manually or not performed at all. That stage is important for this type of choice. At that stage, you may think “What about test pyramids?”.
Test pyramids are widely known models for various usage. They help us understand the different layers of tests we can perform. Additionally, they can help define test automation priorities. One must be careful at that stage as test automation pyramids can be different from generic ones. Test pyramids also lack an essential element, the business priorities.
You can generically say to use the lowest level possible for a particular test. That’s sound reasoning. But don’t forget about the business objectives and your context. One short is to focus extensively on unit tests being faster, more local, etc. But in particular contexts, your priority will be on the user experience, and you may not have a perfect triangle.
This is the point, use models and guides, but don’t forget to see the big picture with common sense. The next activity is to architect the implementation in realistic increments.
Define The Test Automation Methodologies
Delivering activities optimizing our effort can be improved with methodologies. We highlight here the key methods for applying Quality Engineering to Test Automation.
The Agile, Pareto and LEAN methodologies are helpful to consider at that stage. You cannot plan everything, but you can at least architect for uncertainty. A pragmatic approach is to work in time-box increments, prioritizing the value first. The agile and lean rituals can then be leveraged for the planning, issues resolution, and retrospective rituals.
We can already feel ready to start at the next sprint. Stepping back from our test debt experience, we can recall that “it is about the decisions we take and do not take” and the test maintenance issues. To anticipate the effect of delayed feedback loops, we have to clarify the test maintenance effort right from the start. That means concretely allocating time for optimization, rework, and maybe deletion (more on that later) of tests. It must be both defined and limited to contain entropy.
Our test automation architecture is fine but should fit into the overall picture. The next area to architect is the organization.
Architecture The Organization to Support Value Delivery
The test automation tasks will be led to completion by the actors part of a larger system. It is therefore important to ensure that the organizational context will favor the achievements of your objectives. It is about avoiding debt and waste.
The first element is about the Technology Architecture fit. Your work on test automation design focused more on the internal and direct adherences. We need here to step back on the overall IT ecosystem, internal and external, to reduce the risk of integration surprises. We need again to act as a Question Asker as the knowledge is more out of our scope. Is the standard CI/CD pipeline sustainable? What about the scalability of the infrastructure? What are the limitations? Can we perform tests in production? What is the sizing of the staging environments?
These are the type of questions you need to use to validate or invalidate your hypotheses. Changing a plan on paper is relatively easy, even if frustrating. It is much better than when hitting a wall during execution, losing the previous investment. A key practice to maintain the risk level is one of the fast end-to-end iterations. That way, you can quickly verify the transversal integration by setting up reusable foundations earlier. The following organizational part is about people.
The organizational design supports the achievement of a shared mission by a group of actors, known as a team, when collaborating towards common goals. The Conway law clarifies the impact of the organization on the resulting system of collaboration. Architecturing the organization is essential both for ensuring the delivery of specific tasks and supporting the technology architecture we designed. The team can be organized for two optimizations: flow or resources. The flow-oriented is the most adequate for in-sprint test automation that will support fast feedback loops of software changes.
Processes are the prerequisites to effective interactions.
Interactions at the Heart of Organizational Performance
The interactions of the actors also need organization through processes, also known as methodologies. We already cover some of them with Agile and Lean. We identified specific processes required to build test automation wealth. The first one is to define a shared Definition of Done, including specific steps for test automation. It is not necessary to automate every task. The most important is to systematically question ourselves on its pertinence. Complementary questions should be added like maintenance, optimization, or deletion of particular tests.
Culture also impacts the actor’s behavior. We won’t discuss the theme in this article. The general guideline of “test is everyone’s responsibility” is superficial without detailing its rationale and implications. The takeaway is that the various elements we share favor building a shared mission and culture. Responsibilities still require a minimum of affectation for clarity of expectations about roles. The quality of interactions also contributes to the quality of work done. The activities of peer reviews are therefore essential to implement.
The last part that does not come last during implementation is to secure the soft and hard test automation skills. The actual contributors to these tests are the most important ones to focus on first. Secondary stakeholders will have the necessary inputs during the collaborative session while starting. Skills mean that the budget and time must be allocated, ideally with a continuous approach. Learning is about practicing and requires time. It is rarely a one-shot effort. Bypassing this step can result in copy/paste from various online forums, far from being test automation wealth.
The reality can only follow an actual implementation.
Act, Measure and Adapt with Quality Engineering
Delivering test automation with Quality Engineering is an iterative process relying on incremental delivery, measurement, and adaptation. This is where knowing what and how to detect the warning signs of test debt is useful.
The main driver of our actions should remain aligned on the common why and value definition with the stakeholders. Our work is to continuously bridge that gap of value. The value must be measured to ensure the impact of our actions and enable its communication.
Risks and uncertainties are part of the game; there are no shortcuts. Methods are important to follow a structured process. The changing ecosystem is a world of continuous problem-solving we aim to master.
Empowering test automation with Quality Engineering delivers a gap of value commonly defined, where automated tests are part of the overall value creation. It is not about accumulating the maximum number of tests, falling into vanity metrics pitfalls.
By the way, delete non-valuable tests.