Test Analysis Step by Step
In this article, you will get a clear algorithm of test analysis and be able to apply it to real projects, not just for passing a job interview.
The proposed approach will walk you through the test analysis activities step by step and give you an understanding of how to conduct test analysis at different levels.
Test Analysis vs. Test Design
These two terms are often confused and frequently used interchangeably, so let’s begin by discussing the difference between them.
Test Analysis:
- Test analysis is the process of understanding and analyzing the system’s behavior.
- The goal of test analysis is to determine what should be tested, to specify what’s in scope and what’s out of scope.
- It also includes prioritization of the product features based on user needs and business goals.
- It gives us an overview of the system and is a foundation for test design.
Test Design:
- Test design is the process of creating test cases based on the test analysis.
- The goal of test design is to specify how the identified testing scope will be verified.
- Test design activities include:
- Making decisions on test design techniques.
- Creating test cases with clearly defined pass/fail criteria and execution sequences.
- Identifying test data, both input data for the test case as well as data that must exist in the system for the test case to be executed.
- Designing the test environment, and identifying any required infrastructure and tools.
Test Analysis answers what to test, Test Design answers how to test it.
Test Analysis Levels
Test analysis can be conducted at different levels — at the Product (system) level and the Feature level (a specific functionality of the system). The difference lies in the scope and focus of the analysis.
Product Test Analysis:
- The goal is to understand the system’s overall functionalities, their priorities and dependencies.
- It facilitates the development of test plans and test strategies and provides a foundation for regression testing planning.
Feature Test Analysis:
- The goal is to understand the specific feature’s requirements to plan and design targeted tests that verify its correctness and completeness.
- Here we are using test design techniques and creating deliverables for specific features like test cases, test data, and required test environment configurations.
We should start with Product test analysis, which will be a foundation for further analysis of specific features.
Test Analysis at Product Level
There are three main activities of test analysis at the Product level:
- Decomposition — breaking down the system into high-level modules.
- Prioritization — determining the importance of the system’s features based on user needs and business goals.
- Dependency Analysis — identifying relations between features in the system.
Stage 1: Decomposition
The first activity is decomposition, which means breaking down the system into smaller parts. Besides that it helps manage the complexity of software development, it also benefits the testing process in the following ways:
- Increasing our understanding of the system’s behavior.
- Facilitating test planning, estimating, and tracking the progress of the work.
- Giving the capability to evaluate test coverage.
- Providing the ability to perform dependency analysis.
Decomposition can be made at different levels depending on the context and the specific needs of the project. There are decomposition levels like epics, use cases, web pages/screens, modules, objects, etc. These decomposition levels are not mutually exclusive, and they can complement each other. For example, a system may be decomposed into several use cases, and each use case may involve multiple web pages.
A common approach is a decomposition at the object level. Objects represent the entities or concepts in the system that have distinct parameters (properties) and actions (behaviors).
For test analysis at the Product level, we will analyze objects and their actions.
Let’s look at the steps of the product decomposition using the Slack application as an example.
Step 1: Define objects
Here are some objects that we can define:
Can all functionalities in the system be broken down into objects? No, and as was mentioned above, we can combine different decomposition levels. We will use Epics in addition to Objects to describe other functionalities. An epic represents a substantial feature or a collection of related features.
Step 2: Define actions for each object
The next step is defining actions that can be carried out on the object or by the object itself.
Step 3: Define alternative methods for actions
There can be different ways to perform the same action in the system, which should be noted.
Stage 2: Prioritization
Not all features are equally valuable and important for users. Having a list of prioritized functionalities allows us to:
- Start testing more important features, so if there is not enough time to test all we can leave less important features out of the testing scope.
- Find more critical bugs earlier and have enough time to fix them and prevent delays in delivering.
- Create different test suites (for example, a smoke testing suite with critical features, and a regression testing suite with critical and major features).
Step 4: Define priorities for each object/epic
A more advanced approach is to prioritize each action of an object as well.
Stage 3: Dependency Analysis
Most of the features in the system often have relationships with other features. And, the next important step is dependency analysis, which involves identifying the relationships between defined objects/epics, understanding how they interact with each other.
Dependency analysis helps to:
- Identify related features in the system and ensure that all necessary scenarios are covered during testing.
- Identify the potential impact of changes or updates to a specific feature on other related features.
- Assess the potential impact of existing defects on other dependent features and perform targeted testing to mitigate such risks.
Dependency analysis is often documented as a matrix. The matrix consists of rows and columns, where each row and column represents an object (or other item that was used for decomposition). The cells in the matrix indicate the dependency between them.
So, what we need to do, is review our list of objects/epics and think about how they interact with each other, looking for any dependencies such as:
- Functional dependencies — one object relies on the output or behavior of another.
- Data dependencies — one object requires specific data from another one.
Step 5: Define dependencies between objects/epics
Here is an example of the dependency analysis matrix.
This is a one-directional matrix that uses only half of the matrix, where green cells highlight the relationships between items. Additional columns or notes can be used to document the specifics of each dependency.
A more advanced approach is a two-directional matrix that analyzes both sides of the relationships, showing which items depend on others and which items are depended upon by others.
The items at the top in the columns depend on the items at the left in the rows, and accordingly the items in the rows affect the items in the columns. For example, we can see that the Messages object depends on Users, Workspaces, Channels, and affects History and Search.
This type of matrix provides a more comprehensive view with detailed analysis. The choice between one-directional and two-directional matrices depends on the level of detail required for the project and recommended for safety-critical software.
Well, we have done a great job so far and now have a strong foundation for the next level of Test Analysis. Let’s review the algorithm of test analysis at the Product level.
All these activities can be performed in the following subsequence:
- Gather information based on:
- Product documentation (requirements, user stories, user manuals, etc.).
- Product itself (exploratory testing).
- Communication with analysts, developers, stakeholders, etc.
2. Visualize the information
- Different types can be used, such as tables, mind maps, diagrams, etc.
3. Review
- Review the outputs with the team and stakeholders to find missed or incorrect information and make appropriate corrections.
4. Keep it up to date
- As the product evolves, continue to update and maintain the documentation, otherwise it isn’t worth the effort and will get outdated and useless quickly.
Test Analysis at Feature Level
We have learned test analysis for a system in general and now, we are ready to go further and look at test analysis for specific features.
There are three main activities of test analysis at the Feature level:
- Decomposition of the tested object.
- Test Design — creating test cases using appropriate test design techniques.
- Dependency Analysis — defining relations between the tested feature and other features in the system.
Stage 1: Decomposition
For Product test analysis, as the high-level analysis of the system, it’s enough to determine the actions of the object during decomposition, but for the Feature test analysis, we need to proceed further and define the parameters of the tested object.
Parameters are the properties or characteristics of the object, they define the specific conditions, behaviors, or configurations associated with that object. Basically, parameters are values that act as inputs, providing specific information needed for a particular operation.
So here, we have the following sequence of decomposition:
Let’s proceed with Slack as an example and look at the Create Account feature.
Step 1: Define the object and the action
- Object: User
- Action: Create Account
Step 2: Define parameters of the action
To create an account, the user needs to submit a form that has the following parameters:
- Full Name
- Password
- Email Newsletter
Each parameter has requirements regarding what values it should accept:
- Full Name:
- 1–50 characters
- Required - Password:
- 6–30 characters
- Required - Email Newsletter:
- Yes — by default
Next, we need to create test data to check the form submission with different values for each parameter to verify that valid values are accepted and invalid values are rejected.
Step 3: Define test data for each parameter
There is a large number of possible values that can be checked and we usually don’t have so much time to test all possible values for each parameter. Instead, we can explore test design techniques that help to define an efficient set of test data.
Stage 2: Test Design
We move on to the next stage — creating test cases using appropriate test design techniques.
For testing data validation, the following techniques are most often used:
- Equivalence Partitioning
- Boundary Value Analysis
- Pairwise Testing
However, testing input validation is not all verifications that we should perform. Lots of testing that we do involves testing the business logic that sits underneath the user interface. The most known techniques that focus primarily on testing business logic functionality are:
- State Transition Testing
- Decision Tables
We will consider the mentioned test design techniques in detail in the next article.
Test design techniques help reduce the number of tests while maintaining a high level of coverage. However, there are some important notes to remember about test design techniques:
- There is no one perfect technique for all situations. It is important to understand the applicability and drawbacks of each technique and be able to select the most appropriate technique or set of techniques for the testing.
- Test design efforts must be prioritized and balanced to align with the risk levels and business value.
Stage 3: Dependency Analysis
At the Feature level, we also need to conduct dependency analysis, because usually, every new feature interacts with some existing features. Hence, we need to find out these interactions and possible dependencies.
We should take the list of our objects/epics (created at the Product test analysis stage), go one by one, and think if it interacts with the feature we are testing.
Let’s perform a dependency analysis for the “Send Messages” feature in Slack.
Step 1: Find related objects
- Channels: we can send a message to a channel
- Users: we can send a direct message to one user and a group of users
Step 2: Define states of the related objects
- Channels: Active, Archived, Deleted
- Users: Invited, Active, Deactivated
Step 3: Define parameters of the related objects
- Channels: Public, Private
- Users: Regular User, Guest User, External User
Now, we can extend test cases for the “Send Messages” feature with the following verifications:
- Send a message to a channel:
- in different states (Active, Archived, Deleted)
- with a different type (Public, Private) - Send a direct message to one user:
- in different states (Invited, Active, Deactivated)
- with a different type (Regular User, Guest, External User) - Send a direct message to a group of users:
- in different states (Invited, Active, Deactivated)
- with a different type (Regular User, Guest, External User)
Let’s recap the steps of Test Analysis at the Feature level.
Conclusion
We found out that Test Analysis can be conducted at different levels:
- Product level — high-level analysis of the entire product, which is a foundation for further test analysis at the Feature level.
- Feature level — detailed analysis of a specific feature in the product, where we define a testing scope for the feature and design test cases.
Here is a diagram to illustrate test analysis levels and their activities:
Having such a structured and comprehensive test analysis brings the following benefits:
- Ensures a deep understanding of the product.
- Defines a clear and complete scope of testing.
- Ensures that testing efforts are focused on critical functionalities and potential areas of failure.
- Helps to achieve maximum test coverage with minimum redundancy.
- Contributes to defect prevention by identifying issues in requirements.
- Helps to organize the information and reduces the chances of misinterpretation.
It is important to note, that Test Analysis benefits not only the testing team but the whole project team. It encourages collaboration among all team members and test analysis documentation serves as a valuable source for reference, audits, and knowledge transfer within the team.
A well-organized test analysis process provides an adequate level of confidence in the overall effectiveness of the testing and contributes to the delivery of high-quality software.
