Testing Doesn't Improve the Product

Out there in the world, there is a persistent notion that preventing problems early in the software development process will lead to higher-quality products than testing later will. That isn't true.

It's untrue, but not for the reason that might first occur to most people. The issue is not that addressing problems early on is a bad idea. That's usually a really good idea.

The issue is that testing on its own will not lead to higher quality products at all. Problem prevention and testing are different pursuits. Testing cannot prevent problems, and testing cannot improve the product.

Coming from me—a teacher and an advocate for skilled testing—that might seem crazy, but it's true. Weighing yourself doesn't cause you to lose weight. Blood tests don't make you healthier. Standardized tests in schools don't make kids smarter, and certainly don't improve the quality of education.

What testing can do is to improve our understanding and awareness of whatever might be in front of us, by questioning a product in order to evaluate it. Testing—the process of evaluating a product by learning about it through experiencing, exploring, and experimenting—helps us to become aware of problems that might need to be addressed.

In daily life, a particular reading on a bathroom scale might prompt us to eat more carefully, or to get more exercise. A blood test might prompt a doctor to prescribe anti-malarial drugs. Those standardized school tests might suggest changes to the curriculum, or to funding for education, or to teacher training. But until someone takes action, the test only improves awareness of the situation, not the situation itself.

In software development, improvement doesn't happen unlesss someone addresses the problems that testing helps us to discover. Of course, if the problems aren't discovered, improvement is much less likely to happen—and that's why testing is so important. Testing helps us to understand the product we've got, so we can decide whether it's the product we want. Where improvement is necessary, testing reveals the need for improvement.

Some people believe that testing requires us to operate a product, thinking in terms of the product as a built piece of software. That's a very important kind of testing, but it's only one kind of testing, referring to one kind of product.

It can be helpful to consider a more expansive notion of a product as something that someone has produced. This means that testing can be applied to units or components or mockups or prototypes of an application.

And although we might typically call it review, we can apply testing to things people have written, or sketched, or said about a software product that does not yet exist. In these cases, the product is the artifact or the ideas that is represents. Experimentation here consists of thought experiments; exploration applies to the product and to the space, or context, in which it is situated; experiencing the product applies to the process of analysis.

The outcome of the test is the evaluation and learning that happens through these activities. That learning can be applied to the product — but it's the stuff that happens in response to testing, not the testing itself, that improves the product.

Just as testing doesn't improve products, testing doesn't prevent problems either. As testers, we have an abiding faith that there are already problems in anything that we've been asked to test. That is, the problems in the product are there before we encounter them.

So what good is testing if it can't prevent problems? Testing can help us to become aware of problems that are there, whereupon people can make changes to prevent those problems from going any further.

It's a great idea to try to prevent problems early in development. To the degree that the attempt can be successful, we're more likely to develop a high-quality product. Nonetheless, problems can elude even a highly disciplined development process. There are at least two ways to find out if that has happened.

One way is to test the product all the way through. Test the understanding of what the customer really wants, by engaging with a range of customers and learning about what they do. Test the initially fuzzy and ever-sharper vision of the designer, through review and discussion.

Test the code at its smallest units and at every stage of integration, through more review, pairing, static analysis tools, and automated output checking. Check the outputs of the build process for bad or missing components and incorrect settings. These forms of testing are usually not terribly deep—which is a good thing, because deep testing may take time, effort, and preparation that can be disruptive to developers.

In parallel to the developers' testing, assign people to focus on and to perform deep testing. Deep testing is targeted towards rare, hidden, subtle, intermittent, emergent bugs that can get past the speedy, shallow, non-disruptive testing that developers quite reasonably prefer most of the time.

If your problem-prevention and problem-mitigation strategies have been successful, if you've been testing all along, and if you've built testability into the product, you'll have a better understanding of it. You'll also be less likely to encounter shallow problems late in the game. If you don't have to investigate and report those problems, deep testing can be relatively quicker and easier.

If your problem-prevention and problem-mitigation strategies have been unsuccessful, deep testing is one way to find out. The problems that you discover can be addressed; the product can be improved, and problems for the business and for the customer can be prevented before the product ships.

The other way to find out if a problem has eluded your problem prevention processes is to release the product to your otherwise unsuspecting customers, and take the chance that the problems will be both infrequent and insignificant enough that your customers won't suffer much.

When there's the risk of loss, harm, bad feelings, or diminished value for people, it's a good idea to be aware of problems before it's too late, and that's where testing helps. Testing on its own neither prevents problems nor improves the product. But testing does make it possible to anticipate problems that need to be prevented, and testing shines light on the places where the product needs to be improved.