Using Analytics: Why This Is Important for QA Professionals
Using analytics data to help guide your strategy, decisions and focus on your daily activities is key for quality assurance professionals. Especially because understanding the real impact a product is having in the business and in the end user’s life can help you to be more assertive.
How can we concentrate our efforts on what really makes the difference? To do this we need information, so we are not working based on our own assumptions.
Specifically for Quality Assurance Analysts, using Analytics data helps to understand where they should focus the tests on, define the strategy and planning for tests, identify valuable recommendations for the product, avoid spending more time than needed on features that are not so popular when compared to other features.
Moreover, with that data, we can find the foundation for exploratory tests: testing what matters most first.
All of this can help QA Analysts and the entire team to have more confidence in the tests that are being performed for a release.
Valuable information you can get by using analytics
The following topics describe some of the most meaningful information that you as a QA can look into to build a more effective analysis.
1. Most used features
Which are the features that are being most used?
Using analytics to collect this data can turn the team’s attention in the correct direction and shed light on some important questions to focus that are related to those features, like:
- How can we improve those features?
- Are those features providing a good experience for the user?
- How could we make it easier to use?
- How could we engage the users even more?
- What to prioritize in testing on Regression Tests phases?
The same data can bring knowledge about features that are not being used or don’t have significant usage. Some other important questions that can be raised to be analyzed when considering this data are:
- What is causing this feature to not be used as we expected?
- Can the feature be easily discovered?
- Is this feature delivering something meaningful for the users?
- Is the value delivered by this feature clear enough for the users?
- Is it working properly?
- Is it providing a great user experience?
- Is there something we can improve on it?
- Should we keep this feature or remove (maybe replace) it?
Besides all questions described, this information can also help QA Analysts to define the test strategy, the prioritization and the scope for tests. For example: For a Regression Tests scope, a bigger percentage of time and effort could be spent on the most used and most popular features along with the new ones that are being released, while a smaller percentage of those resources would be used on the less used ones.
This relates to something that we insist on: look at the big picture — understand what we are delivering and the impact it has.
2. Paths commonly followed by users
For users to complete a task in an app, it takes a path. Sometimes, more than one path is possible. Knowing the journey most commonly chosen or naturally followed by the users and if the tasks are successfully completed or not is useful information to understand user behavior and what can be or needs to be changed or adjusted in the application.
Writing test scenarios that replicate those paths and include them in the test cycles for each release is a good way to use this data. Using analytics can also be valuable when an exploratory test is needed for any reason, as it allows us to ensure those paths are being covered in tests. So don’t forget to keep an eye on this data since it can change along the way, so the test scenarios should be adjusted as well.
In addition, this type of data could help your team to evaluate if a specific available but unused path should be removed and/or therefore no longer receive maintenance.
3. Impact of bugs in live apps
When a bug is found, it’s important to evaluate not only the problem itself, but how it will impact the end-user and what is the size of this impact.
Important data to collect when using analytics with this purpose are:
- How much is this feature (where the bug happens) used?
- Is this issue something that blocks the user from using it at all or it causes a setback but the user can still complete the task successfully?
- Is this happening on all OS versions/all platforms? If not, try to collect information such as the number of users using that specific feature and the amount of those users in the specific version where the issue happens.
- Would that device or OS version be an option the team should consider including in the project device matrix if it’s not yet included?
We know that when it comes to the matter of product development, many variables compose this “universe” including scope, time, effort and budget. Considering all those variables, the team should seek to be more efficient and more agile every day — focusing on delivering the best results.
A bug that happens on a popular feature probably should be highly prioritized to be fixed. On the other hand, a bug that happens on a not so popular feature may be moved to the end of the list. Of course, the severity of the problem should be taken into consideration as well for this decision. However, this prioritization is important considering the time factor. Imagine your team is really close to the finish line and is also running out of time — in such situation, assertively deciding on which feature to focus on is essential.
Here’s a more detailed example:
Imagine a problem that happens on a feature that does not prevent the user from using it and completing the desired task. The bug is bad, but the feature is only used by a very small percentage of active users. In addition, that problem only happens in a specific device model or OS version that is one of the less used as well. With that in mind, there are other more important tasks and with higher priority to be developed, also, there are other bugs that are affecting very popular features that have a high number of active users. In this scenario, the latter bug would probably be placed after the other priorities.
Note that by using analytics to favor data driven prioritization, the QA will have valuable information to bring to the team when informing about that bug, so the collective decision can be more assertive.
4. Crashes
We all can agree that this is information that may cause some stress on the development team, but it can also be really useful. Following up on Crashlytics is an important task that the team should accomplish. Especially because if crashes are happening, there is a big chance they will need a fix ASAP!
Of course, this will also require further analysis that can include:
- % of crashes happening compared to the total number of users
- The version where the crash is happening
- Device/platform/ browser / OS version where it’s happening
- Understand if it’s an edge case or not
All this information obtained by using analytics can help you to reduce the scope to investigate the issue and also increase the chances of discovering the causes as well as providing a solution faster.
5. Versions adoption
Knowing the app version adoption as well as the percentage of people who use each of them helps in the analysis of which versions should be prioritized when testing the version update.
For example: If we know how many users are still on version “X”, and how many are on version “Y”, we can easily define which versions we should check to confirm that no issues will happen when updating from version “X” to the new version “Z”.
6. Most used platforms, device models, and OS versions
When a new product is going to be developed, we need to know how it will be covered in tests.
This includes knowing which platforms, devices, OS, and browser versions will be a part of it. This information can be obtained from analytics from the specific client or, if not available, from generic research filtered by the region where the product will be available. In this case, is also important to check:
- What are the most used device models x OS versions
- If users prefer desktop, mobile or use both.
With all the possibilities using analytics provides for quality assurance analysts, there’s no wonder why it should become a part of these professionals routines. Here at ArcTouch, for example, it already is. (By the way, if you would like to work at a place where quality assurance is seen through a data-driven perspective, you should really subscribe to our job alert).
