Test Strategy Storming
A lightweight method to get a Minimum Viable Test Strategy
If one can get a good overview of a complete domain with a simple Event Storming workshop, why not do something similar when you want to design a test strategy? This article presents a new collaborative approach to develop an agreed picture about how to get started with testing and about open points that needed to be clarified on the way.
Motivation: Why do Test Strategy Storming?
In classical approaches a test manager takes the first weeks of a project to write an extensive test concept / test strategy based on early defined functional and non-functional requirements. This was often followed by a longer review and acceptance process by different stakeholders.
Agile environments need a different approach. Requirements are not stable or not even known at the beginning. We expect and want them to be changeable. Teams want to start to develop features from day 1 and those features are supposed to be in good enough quality to be deployed in production when they are finished. In contrast to classical approaches. there should not be the need for long late test phases. So, it is essential, that teams have a common understanding how to do testing right from the beginning. Like we build minimum viable products in a short time, we need a quick way to get a minimum viable test strategy that can develop over time.
Method: How to do Test Strategy Storming?
Analogy to Event Storming
Event Storming has proven to be an efficient way for generating a common understanding about complex processes in a short time frame. So, when we had the situation in a project where we needed an agreement about a test strategy very fast, the idea was born to copy some of the ingredients from event storming and adapt them to the topic of testing. Like in event storming, the approach is to:
- Bring the right people together
- Focus on a common understanding
- Make it informal and lightweight
- Visualize the outcome
Topics for Test Strategy Storming
The topics to be covered are typical contents of a test strategy:
- Test stages (with their respective test objects)
- Test types (for the ISO criteria to be covered)
- Roles & responsibilities
- Metrics and quality gates (including test coverage)
- Test tools
- Test environments
Test stages and test types span the landscape on which testing is performed. At least the rough shape needs to be clear to successfully distribute testing activities. The more complete the view on test stages and test types is at the beginning the smaller is the risk that test activities are started too late, which often leads to the unnoticed piling up of technical debt.
When the areas on the landscape are defined, roles need to be assigned who are responsible to shape and execute the testing for the different areas.
Like architectures evolve in agile working models, there should be room for evolution in testing. So, we rather want to define metrics that indicate a successful test instead of too many details how to get there. Only agreements that need to be shared by multiple teams and that need some central governance to ensure efficiency must be addressed early. That’s why test tools and test environments should be topics early on.
Minimum Viable Test Strategy as Outcome
The outcome is enough common understanding among teams and relevant stakeholders to get started with testing, not a complete gapless test strategy. Something that we call a Minimum Viable Test Strategy. The testing defined by this strategy should be good enough, so that the teams do not pile up technical debt constantly sprint by sprint because of missing quality assurance.
A list of open points is considered an essential part of the outcome. They form an initial backlog for further detailing the test strategy.
Workshop Participants
Like for event storming, the common understanding is to be reached in a workshop with “the right people”. Who those people are, can be very different depending on the organization. For a small project, it can simply be the development team with product owner and someone with test knowhow from the organization. For larger organizations, typical candidates are:
- Representatives of the development teams
- Product owners
- Architects
- Testers
- Test- or quality managers
- Security experts
- Operations
- Experts for build infrastructure (e.g., from a platform team)
- Project managers
Visual Representation
To make the workshop lightweight, no preparation by the participants is needed. Nevertheless, we want to make sure that the relevant topics are covered and visualized to make sure that there really is a common understanding.
The core structure is a table with test stages in the rows and test types in the columns. If a test type is planned for a test stage the corresponding cell is filled with different colored post-its that represent roles, tools metrics and environments. Post-its for open points can be added anywhere.
In a simple context, the fully filled table could look like this:
Workshop Flow
To summarize, the only preparation consists of:
- inviting the right people
- preparing a brown paper/ whiteboard or virtual equivalent with a large (empty) table
- providing multi-colored post-its
The complete workshop can roughly be split in three parts:
- Test stages & test types
- Specifics of functional testing
- Specifics of other test types
In a virtual setting, you might want to split it in 3 different meetings. The time that is needed highly depends on the how clear the picture is among the participants and how complex the setup is. E.g., more non-functional requirements mean more test types, which often means more discussions. In a complex unclear environment, the workshop can easily fill a complete day.
The goal is to have common understanding of the complete testing landscape. The way to get there can be different. So, feel free to experiment. We have good experiences with the following flow:
- Start with an introduction about the goal of the workshop and the visualization. It might be helpful to use the table cell “Functional Unit Test” as an example for how the outcome can look like.
- Work with a moderator who guides through the topics. It is recommended to have a moderator with testing knowledge, e.g. an experienced test manager, who can start discussions, challenge agreements and find gaps.
- Start with defining test stages, then test types. Don’t cover the table-cell-contents until stages and types are clear.
- If a test stage or type is out of scope for the group in the workshop, it is worth to show it in the picture as out of scope to make it transparent.
- For the filling the table-cells, start with functional testing and define the specifics test stage by test stage.
- Then do the same test type by test type. You will not have each test stage for each test type. Just mark not applicable combinations by a cross.
Some hints from practical experience:
- Compared to event storming, there is less “storming”, i.e. moments where everyone adds content in a brain storming way. So, participants may be less involved. Motivate the participants to write the post-its when they suggest something.
- Don’t put much time in theoretical discussions like what is a test type vs. a test stage. If the audience agrees on something that does not follow official definitions, accept it. It is about common understanding after all. An example can be Consumer Driven Contract tests, which we accepted as test stage, although it better fits as a test type.
- Challenge fast agreements. You might think long discussions are the biggest risk, but no discussions and dropping topics can be even worse. For example, an agreement to do unit testing with Junit and the typical coverage metrics might be reached fast. But you might have a web client that cannot be covered by Junit. Shouldn’t there be unit tests as well? And what about static code analysis? Do we consider this as a unit test, or do we have an extra test type for maintainability? Whatever is decided, there should be metrics for static tests as well.
- Find you own way. The method described here is not a corset. If a certain part bothers you, if there are other information that you need at the start, just change it for your workshop.
- At the beginning of an agile project, you often don’t have a deep view on non-functional requirements which are the basis for non-functional tests. Test strategy storming will make this visible. In many cases this is acceptable. In these cases, just take the conscious decision to move the topics to a later point in time. For the other cases, you get the benefit of having the discussion early enough to take actions right from the start.
- You can use the tabular representation of a testing landscape not only for storming workshops to define a new test strategy like it is described here. It can also help to visualize existing test strategies, for example if you want to discuss problems, improvements or misunderstandings.
Possible extensions:
- Test environments can be complex and controversary topic, especially if different responsible parties and multiple systems are involved. If you think just naming the environments is not enough for a start and you want to clarify more details already in the workshop, you can add an extra table for test environments. There you can collect specifics like owner, test data, integrated systems and required mocks for each needed environment.
- If you want to put an early focus on test data, you could add an extra card type (i.e. post-it color) for the topic. On the cards you could state the owner of the data, but also details whether synthetic data are to be used or masked data from production.
- It is usually well perceived that participants can join without necessary preparation. Having the right people together should ensure that the knowledge about given rules or requirements like contractual agreements or company standards is present at the workshop. If not, send out related documents in advance or add a short introduction session at the beginning of the workshop.
- If you fear that people might not be engaged enough, consider standard moderation techniques like icebreakers at the beginning or after breaks.
Summary: What to expect from Test Strategy Storming?
At the end of your test strategy storming workshop, you have a common understanding about the testing landscape, i.e. the planned scope of testing activities. Additionally, there are enough details for teams to get started with good enough testing for the upcoming sprints. The visualization will help communicating the scope and expectations to teams or other stakeholders.
You will not have answered all questions around testing. Don’t try to solve all problems, it is enough to make them visible and get an understanding about priorities. An important outcome is a list of open points and tasks that form an initial testing backlog. Your next challenge is to find the time and people to constantly work on these tasks and to decide about new testing related topics that come up over time. With test strategy storming you do not have completed the task “create test strategy”. It’s an ongoing effort. What you have is a reduced risk of forgetting important aspects or running in a wrong direction and producing a huge mountain of technical debt.
The approach of starting with a minimum viable solution and then adapting it step by step fits very well in an agile environment, but can also be used as a starting point in classical contexts. For people with a classical testing background, it means a mindset change in any case. Therefore, expectation management is crucial.
