What is in your Testing Scope?
I decided to talk about the testing scope since I realized that not everyone clearly understands what it is and how to define it. What should be included and what should be ignored? How detailed should the scope be? Do I have to align it with my stakeholders? What if the deadline is coming, and I don’t have enough time to test even the business-critical areas?
Before starting our journey into the testing scope depths, I would like to mention that the testing scope it’s not a “must-have” as a QA base skill set. I chose to use it because I found it fits the best to my testing needs. In this article I’m sharing my personal thoughts and the vision of the testing scope.
Our role as testers tells us that in order to gain the accurate assessment of a product’s quality we have to test simply everything. In an ideal world that would be the case. However, as our reality constantly changes, the best we can do in order to provide a result without losing those important moments is to perform short test cycles with fast feedback about the state of the product.
But how can I deliver a quick result if, potentially, I have an endless list for testing, you may ask. The answer is to create a testing scope that focuses on the most critical product aspects based on the business needs.
Before we discuss how to design the testing scope, let’s first define what it is.
A Testing Scope is a list of product features, product parts, or product-related integrations that must be tested in order to build a reliable assessment of a product’s quality.
The Testing Scope doesn’t include details of the testing strategy, the automation approach or the tools that are going to be used. I won’t list who will perform the testing or when it will happen. The document that represents the details above is called the Testing Plan. It answers the questions “Who”, “When”, “Why”:
- who will perform the testing,
- when to perform the testing,
- why to test / not to test specific product parts, etc.
The Testing Scope itself answers the question “What”: what to test.
Let’s take a look at a small example. Imagine we have a feature, ‘a new layout for a site component’. Let’s think about which parts of that feature should be tested in order to conclude whether it meets the expectations or not.
In the example above you can find the different testing types like UI, functional, performance, accessibility. Each product item is defined on a very high level.
How to find out which testing types to apply?
To answer this question, explore your feature from different angles:
- feature has a UI part — think of adding UI/UX, functional, accessibility, performance testing to your testing scope;
- feature uses an API, it makes sense to test server-side validation and error handling;
- feature reads/writes to a database — security testing helps ensure there won’t be unaccounted DB access for malicious reasons;
- your product has a representation on mobile devices — mobile testing will come in handy.
I prefer to talk to the product manager and the developers before starting to work on the testing scope. This gives me the most up-to-date information. Documentation usually is written in the early stage and not all the changes that happen during the development are reflected there.
After I gather the information, there is a good practise in my team to do a collective brainstorming session. I work with 9 people, great experts, whom I can invite to this session where we think of any possible user scenarios and examine failure cases. The result of such a session is a draft of the testing scope that later on will be systematized and optimized according to actual resources that I have.
We don’t do brainstorming for every single feature. For small features usually it’s clear what has to be tested. But for large complex projects with a lot of integrations and dependencies, having a second pair of eyes helps a lot. At this stage, when you just explore your product or feature, there is no need to worry whether you have enough time to test everything you plan or not. Be agnostic to your environment, try to produce as many ideas for testing as possible and later on do another talk with your developer and/or product manager to figure out what is critical and what can be skipped.
What should a testing scope level of detail be?
The level of detail of the testing scope should be selected according to the needs of the tester. For instance, If I write the testing scope only for myself (I’ll be the only QA on the project), the left part from the example above should be enough for me to understand the task. In case I create the scope for a group of testers that are new to the product, I will most likely add more explanations and even split each item into subtasks.
It is possible to list the feature parts that won’t be tested in the Testing Scope. Or to consider that everything that is not listed in testing scope (out-of-testing-scope) will be omitted during testing.
Which stakeholders should be aware of the testing scope?
There are several reasons that may incline you to notify some of the stakeholders and ask them to review or approve your testing scope:
- When a feature has parts that for some reason cannot be tested, it’s highly recommended to let the manager and the developer know. It always leads to some risks that should be discussed and accepted;
- When you find the use case scenarios that are not covered in the design or the documentation, it makes sense to raise an issue about it. In my practise designers might forget to add an error screen (general error like 404 or 500 or a specific server-side error);
- Sometimes, when working on the test scope you may find the illogical user scenario or the missing requirements. And again, it’s better to raise this issue at the early stage, rather than after the development is completed.
Since we defined what the testing scope is, let’s consider what can help us optimize its content and focus on the very critical areas for the business.
How does the business influence the testing scope?
Let’s examine particular products and features according to their business goals and figure out how they affect the testing scope:
- Killer Feature or a Game Changer: an innovative feature that potentially can take your product to market leadership. In this case time plays the most important role. The sooner the killer feature is released, the more users it engages. Quality becomes secondary: once the happy flow is working, it’s acceptable to release the feature to the customers even with a large number of non-critical bugs. For testers such business goal means performing frequent test cycles focusing on the critical business areas;
- Refactoring of a critical business area. Refactoring means that the user experience should not change. The goal of such a job, in most cases, is to improve performance, strengthen security, migrate to a new API, and so on. Thus, the task for the tester is to prevent regression. If during the release the main feature flows work as expected, but we produce a large number of new bugs, the users most likely will be unhappy and may tend to leave our product. In the testing scope, it makes sense to thoroughly cover each product part with tests and, in addition, add a scenario that covers the feature’s intent: execute performance testing, verify new API, examine security;
- PoC (Proof of Concept). PoC is an idea or a solution that the company wants to try and explore before releasing to a large audience. PoC can also be used to choose the best from several similar approaches or technologies. The main goal of a PoC product is to demonstrate the particular solution, no matter how buggy it is. Often such a feature doesn’t require any testing at all, but in particular cases, testers may be involved as domain experts. In this matter the testing scope should include use cases that help reveal the weak spots of the solution chosen.
Examples above illustrate how the business changes the testing focus and directly defines the testing scope. That’s why it’s important to ask WHY: why we want to release a particular product, which user issue it aims to solve, how it helps the company to achieve its goals. Many times I noticed testers that while working on a big project weren’t able to answer a simple question — ‘why this project and why now?’. Not knowing Why creates a risk of missing the deadline (if you are working on a killer feature), or doing extra work (if it’s a PoC approach).
How can people in your team help minimise your testing scope?
What you should definitely avoid while testing is performing double work. If you have a UX designer in your team that always verifies the user interface before a feature release, you can use it as a chance to shorten your testing scope. Of course, the designer can miss some client issues (nobody does testing better than you), but that is the risk, the justified risk.
As testers we cannot afford ourselves to take enough time to cover every single piece of our product. We are limited in time, resources and budget. That’s why we have to manage our assets carefully. Thus, it’s good to know our team members, what they are doing and not to be afraid of delegating a part of our responsibilities. Of course, if that doesn’t contradict with people’s roles (simply, don’t expect a business analyst to be able to perform proper API testing :)).
In a situation when you have a tough deadline and you don’t have the time to cover even the critical product areas, it might be a good idea to ask your team for help. Do a bug hunt session, take this chance to learn how the people you are working with explore the product, what is important to them in the first place.
Conclusion
The testing scope is a high-level list of product parts that have to be tested in order to gain a reliable assessment of a product’s quality.
To figure out which testing types to apply, explore your product from different angles. Choose the level of details that fits your needs best: think who will perform the testing, how well the executors know the SUT (System Under Testing).
As an additional step, check if your testing scope has to be reviewed by your stakeholders. This helps align the deadlines and clarify all the controversial points.
Finding out the business goal of your product or feature will give you a clue about what to focus on. Talk to the owner of the product, get to know why the company wants this feature and who will be its audience.
And finally, consider delegating some parts of testing to your teammates. Often we are limited in time or human resources, and it’s totally OK to ask for help and collaborate to achieve the one goal — ensure the product’s quality.
