Why Quality Isn’t Equal to Testing: Where Quality of Your Product Truly Resides?
I believe that the statement ‘testing is not quality’ is one of the basic fundamentals of software quality. It is not just another theoretical statement for someone to pass an interview. Disregarding this principle can cost you a lot.
In this article, I will attempt to explore this statement and provide comprehensive examples from my experiences.
Usually, during the common software development lifecycle, when discussing whether a feature meets the expected quality, teams typically focus on testing: unit testing, integration testing, acceptance testing, and so on. At the beginning of the product’s lifecycle, when the company or team is newly created, this approach works well, allowing you to catch bugs and deliver a product of expected quality. You diligently test all your features because there are not many of them. You create all necessary test artifacts: test plans, test cases, edge test cases. You employ appropriate test design techniques and ensure everything is automated.
However, as the number of features increases, so does their complexity. Stakeholders want to add more integrations, and regression testing becomes necessary. The company decides to hire more QA engineers with impressive experience and outstanding automation knowledge. But here’s the catch: the dependency is not linear. If the number of features doubles, it doesn’t mean hiring just one more QA engineer will cover the lack of tests and address the issue of poor quality.
The paradox lies in the fact that the more engineers you hire for one team, the more entropy you introduce. QA engineers spend a considerable amount of time communicating with the team, generating code, maintaining old code, reviewing specifications and test cases. As a result, the same number of bugs or even more may be missed.
Am I suggesting you should not hire new QA engineers? Absolutely not. What I am suggesting, and will further develop later on, is that the company or team management should do it smartly, basing their decisions on an understanding of the quality of their product.
To provide more accurate points, I’d like to offer a real-life example. Imagine you have a house with several issues: the entrance door lock is broken, one of the light bulbs in the ceiling has burned out, and you need a new set of gardening tools. However, you only have limited resources amounting to $150.
The simplest approach might be to divide the $150 equally among the three needs, allocating $50 to each. However, such a decision may not yield satisfactory results: while you may have a functioning light, the cheap lock may not provide adequate security, and the gardening tools within your budget may lack essential items.
A wiser approach would be to prioritize the most crucial issue. For instance, allocating $100 to fixing the door lock, especially if you live in an unsafe neighborhood, would be prudent. The remaining $50 could be saved for future expenses. As for the gardening tools, you could consider requesting them as a birthday gift, thereby reducing immediate financial strain
What I am saying here is that when you focus solely on methods, problems, and solutions without considering the whole picture or understanding how problems arise, you will not effectively allocate your resources.
Returning to the context of software development, I want to emphasize a similar situation with testing. When a team lacks a comprehensive understanding of the overall project goals and expected product quality, its engineers expend resources blindly. They diligently follow formal testing procedures, such as testing specifications and creating test cases using various test-design techniques, and strive to automate processes extensively, often at significant cost.
Well, here you’re probably wondering, what exactly it means to see the big picture. Let me break it down:
Know the product
- Who are your customers?
- What are the most common usage scenarios?
- What are the most profitable scenarios for your product?
- Are there any overlaps between these two?
- Are there any third-party integrations? Which ones are the most fragile or risky?
- What are the most crucial parts of the product?
- What are the less crucial parts of the product?
- Where in the codebase or microservices are these crucial parts located?
- What infrastructure do you have underneath? (VMs, Kubernetes, cache engine, database)
Answers to these questions should not only be formal knowledge. All of them should be taken into consideration when planning testing and allocating resources for particular cases.
Know the value
The team should understand every scenario and every microservice that brings value for non-commercial organisations and generates profit for commercial ones. Let me provide the following example: the company produces software for cash machines. Currently, the software can perform two operations that bring the majority of the value: cash withdrawal and invoice payment. There are two types of machine manufacturers: type A and type B. Type A machines are mostly located at gas stations and malls, while type B machines are mostly found in universities and hospitals. How would we compose test data without knowing the value of our software? Well, we might validate happy paths for each type and operation. Then, let’s say we create 10 negative cases (based on QA team capacity): 5 for cash withdrawal and 5 for invoice payment. We will alternate the type of machines as test input data, with 3 cases for type A and 3 cases for type B. At this point, we might think we have uniform coverage of our features. But is this what we really need? Consider the consequences of invoice payment not working at a gas station — perhaps not dramatic. However, the consequences of a malfunctioning cash withdrawal option at a remote gas station could lead to customer frustration and anger. In contrast, if invoice payment were out of service in a hospital or university, it could paralyse the entire organisation's operations. Returning to test planning and leveraging our insights, what should our test plan look like? For example:
- N happy paths for each type of machine for each type of operation
- 1 negative test case for cash withdrawal on machine type B
- 1 negative test case for invoice payment on machine type A
- 4 negative test cases for cash withdrawal on machine type A
- 4 negative test cases for invoice payment on machine type B
With this approach, allocating the same resources and having the same limitations, we would actually have more coverage of crucial functionality, mitigating the risk of dramatic consequences in case of failure.
Please be aware that this is a simplified abstract case. Surely, we can debate on this approach, implement simple automation, and find better ways to test everything. The purpose of this example is to illustrate how to implement a smart approach in your testing, maximizing its effectiveness.
Know your weaknesses
Countless times, I have heard from teams when investigating a bug: ‘Oh, it’s a known problem, just restart that service.’ Or implementing features that will add load to the database, while the database instance has the smallest type in AWS. Every system, natural or artificial, has weaknesses — it’s okay. What’s not okay is not knowing where your weaknesses lie. Do you remember what happened with the seemingly impregnable fortress in Helm’s Deep when the Rohirrim forgot they had a sewer hole in the wall? They encountered a fatal problem. So the team should have an absolutely fair list of all their weaknesses, including all bottlenecks and bus factors. Here is a brief example of what may be included in this list:
- Load on the database, which is hard to handle due to poor architecture or lack of infrastructure
- Concurrency issues such as deadlocks
- Complex integrations with other systems that are not robust
- Specific features for your app like timezone-sensitive functionality. I worked on two projects where the core functionality was time sensitive. In one project, all bugs found in this area were considered out of scope as there were no scenarios where an event in timezone A would be created from timezone B. However, in the rare case when a bug did occur, the fix was not quick. It took time to create a solution that would not affect the current logic working with time and dates. Surprisingly (or not), timezone issues regularly appeared. And as I mentioned, the fix was not easy because it required additional analysis each time the problem occurred, which was a real pain. In comparison, in my second project, the team was aware that this could possibly lead to big problems; they knew it was a weak area. But at the early stage when we were planning the MVP and didn’t know all potential usage scenarios, we decided not to spend a lot of resources on fixing potential problems. When these bugs appeared some months later, the fix was quite quick because we knew our weaknesses and responded accordingly
To sum up, there should be no gray zone in your system. Even if everything somewhere is dramatically bad, you should know it before your customer (or attacker) does.
Know the risks
This point builds upon all the previous points. Frankly, I couldn’t express it better than Tom DeMarco and Timothy Lister in their book ‘Waltzing with Bears’. This book is brimming with practical advice, real-life examples, and guides to implement in your project. In essence, risk is not something from the realm of fortune-telling; it is decomposable and measurable. So, I highly recommend reading this book or searching for resources on creating a risk matrix for software projects online — though the book is preferable. How does understanding risk affect your quality assurance process? It’s mostly a reverse process. As an engineer responsible for QA and testing, ensure that all quality risks are included in the risk matrix.
Smart Test Automation
Honestly, this is my favorite point. I don’t know why, but there seem to be two prevailing strategies: either zero automation or extensive automation for the sake of automation (mostly e2e). There’s nothing in between. Allow me to share an unpopular opinion: automation is a tool used to achieve goals in the most productive way possible. Automation can’t be the goal itself. When it becomes the goal, no one cares about quality anymore. The QA team ends up with another huge codebase, with a large legacy part, solely focused on achieving a 100% pass rate and spending time until lunch investigating failures from overnight test runs. I could write three more articles sharing my thoughts on why I don’t like verbose automation. But let’s focus on what smart automation looks like:
- E2E test automation is very expensive: it is resource-consuming during creation and maintenance stages, time-consuming during execution, and prone to errors due to numerous dependencies. Therefore, it’s advisable to keep the number of E2E tests to a minimum
- Consider E2E tests as acceptance tests. Make all assertions at once using soft assertions or fixtures with specific pre-steps. Dispose of tests where you make multiple API calls and perform a SELECT query to the database solely to validate that the response field ‘name’ is a combination of the name and surname initially sent to the backend
- Configure integration tests for your application. Utilise the old good integration tests with test containers and mocks. The configuration can be resource-intensive, but it is 100% worth the effort. Here, you can incorporate the highest number of tests at an affordable price
- Focus most of your efforts on creating test scenarios. Decompose them and decide at which level or combination of levels they should be automated: integration or end-to-end. Because, in the first place, you should always know WHAT to test, and only then HOW to test
- Delete all flaky tests. Remove or reconfigure tests with dozens of retries, especially if they never find bugs. They will always bring more problems than benefits
- Common sense. Here, I would include all well-known statements such as UI tests should be automated using a lightweight programming language, the code itself should be comprehensive, and tests should be independent
- Automation is more than just tests. What? Okay, have you ever been in a situation where a team member wants to check something manually on the test environment? They need to generate test data to play with it. This team member comes to the QA engineer, who runs a test in debug mode, retrieves the ID of the transaction or name of the customer, provides it to the team member, and then goes back to their tasks. Now, imagine what if instead of ten automated tests that never find bugs, a QA engineer creates a tool which generates appropriate data. So, you can have all your team members involved in the testing process, no one actually tightly coupled to a mysterious autotest framework and QA working hours. Everyone can test everything at any time. And also, this tool can be maintained not only by QA but also by developers. Trust me, it is a brilliant approach. I’ve used it a couple of times and it really helps not only during testing but also during the analysis stage. I don’t know why it is not very popular.
Analyse bugs
Divide your product into areas: by architecture, by business logic or by importance. The more, the better. Then, label these areas. Apply these labels to each and every bug as well as to the automated tests at each level. Conduct regular analysis to answer questions such as where do we have the majority of bugs? Why did the tests with the same label miss them? Then, improve all necessary processes and areas in your team accordingly.
Take out metrics from the grey zone
Oh, we love metrics, don’t we? The more, the better… or is it? Well, metrics are helpful and powerful tools. Unfortunately, they are often used like a Fairy Godmother: you put as many metrics as possible on your dashboard hoping that when something goes wrong, the magic will happen and these metrics will help you understand the problem. I promise, magic will never happen. To be in control of the quality, everyone on the team should understand what each metric means and which ones are important, and what value each particular metric brings to your product.
Try this experiment: go ask your teammates what P95 means? What is P99? P50? Is the explanation comprehensive or just a Wikipedia summary? Okay, why are you choosing one over another for your services?
So, the most important thing about metrics is to train every team member to understand the metrics and see correlations between metrics and the quality of the product. The second point is to get rid of the useless ones. Add metrics only when you feel you need them.
Support your support
This point is definitely optional, but in my opinion, the quality of the service is not only about how good your service is, but also about how well your customers are treated if something goes wrong. To provide your clients with the best user experience, train your support team as James Bond trained for his duty. They should know all the points mentioned above (except maybe test automation), and they should always stay updated.
Okay, well, wow. I thought it would be a small article — a checklist, but it turned out to be a draft for a book. The thoughts I have shared with you here are not definitely the one and only source of truth, but they were born from my experience. They address the problems we faced and how we solved them, or how we managed unexpected shortages of QA engineers and still provided the expected quality as a team.
I encourage you to review your product with all these points, and I bet you will gain a lot of insights. And please remember once again: Quality is not just testing. Quality is much more than testing.
