Testing Requires Experience

Testing requires experience, but maybe not in the way that comes to mind immediately. To clear that up, maybe it would be better to say that testing requires experiencing.

Time, then, for a few words on experiential testing, a positive replacement for one aspect of work that people haplessly call "manual testing". In the Rapid Software Testing namespace, testing is experiential to the degree that the tester's encounter with the product is practically indistinguishable from that of the contemplated user. 

Highly experiential testing typically involves direct, naturalistic interaction with the product as some contemplated human would normally interact with it. Experiential testing is important, because much of what goes on in testing may be instrumented; not direct, not the way real people really use or interact with the product. Testing-wise, instrumented testing is both feature and bug.

A key difference between medicine and poison is dosage. Instrumented testing can be feature, medicine, super-useful when we want to extend or accelerate or enhance some aspect of what we can to operate or observe the product. Instrumentation allows us to enable behaviors that are hard or slow to perform by normal, human interaction with the product. This allows us to learn things product behaviours that would be impossible to learn by other means. By this reckoning, experiential testing is bug, not feature.

Yet instrumented testing can be seen as toxic, bug, because it alters and distorts and displaces various aspects of how real human beings really experience the product in real time. In the end, the product is only a medium between people, their relationships, their tasks, and their goals, So remember: instrumented testing is both bug and feature at the same time. 

The trouble I see is that these days, there is nearly unbounded enthusiasm for instrumented testing, and experiential testing gets short shrift. It seems to me people pay rather too much attention to the bug perspectives on experiential testing and not enough to its features. 

Experiential testing is often described as "slow", and "error prone", as though performing a bunch of simple checks at lightning speed were the only goal of testing, or that instrumented testing is somehow immune from error. Automated processes allow us to do excellent testing work quickly They also allow us to do bad testing faster and worse than we've ever done it before.

Instrumented testing often leans in the direction of the builders' ideas about the product and problems in it. In the end, it's people who are going to be using the product, so in testing a healthy chunk of our encounter with the product should be as its users encounter it. 

Now, you might say "Experiential testing is about using the product like some user might use it. So isn't experiential testing just 'usability testing'?" I'd reply No. When we're performing experiential testing, we might reveal usability problems. Yet experiential testing can be focused on all kinds of different quality criteria and risks that users might encounter. Experiential testing is about gaining experience experience with the product. 

Interacting with the product naturalistically is an essential aspect of learning about it, understanding how people might obtain value from it; identifying risk areas where value might be threatened; finding out who might be bugged by problems; and learning about how we might test the product in deeper, richer ways. 

When experiential testing is happening, a casual observer might find it hard to distinguish between the tester's actions and the actions of the user the tester has in mind. The differences lie in the intentions and the responses to what happens.

A experiential tester will be using the product like a user to drive discovery, investigation, and learning; the contemplated user would simply be trying to get work done. The tester will be actively looking for problems; the contemplated user would simply be suffering from them. The tester will investigate and report those problems to the development group; the contemplated user will more likely curse and move on.

Experiential testing doesn't necessarily involve the GUI. The tester, contemplating a user, will use the interfaces that that contemplated user normally would. Of course, typical end users do use the GUI to get their work done. Some users use the product's wizards or built-in macro facilities to accelerate their work.

People often take advantage of a product's API to perform instrumented testing, and that's fine as far as it goes. Testing through the API affords lots of opportunities for high-speed, high-volume transactions, lots of automated checks, and lots of variations in the data (although, sadly, it seems that many opportunities get missed for using APIs to support exploration).

Yet it seems easy for some testers to forget that some users (we call them "developers") use APIs to get things done with the product. Thus testing through the API can be experiential when the tester tries to use the API the way a contemplated programmer might use it.

By all means use the API to instrument and probe the product, but also consider trying to build something with it, using the API and its documentation. That kind of experiential testing can reveal problems that a developer might encounter: confusion about the interfaces, inconsistencies between the API and the documentation, missing features or functionality...

Experiential testing also happens when we encounter some aspect of the product as, say, its ops person might; or as its tech support person might; or as a documenter might, all using their usual interfaces and tools in normal ways when we consider them as users of the product. Testing is also experiential when a tester encounters the product as a tester does—which we might do in a deliberate attempt to expose testability problems.

Experiential testing might sound similar to exploratory or interactive testing, but they're not the same. Testing is exploratory to the degree that the tester, rather than some outside influence, is in control of the testing work. Testing can be scripted yet experiential.

For instance: when a tester goes through the user tutorial, very deliberately, step by step, the testing is experiential and yet highly scripted. Testing can be highly exploratory yet not experiential—for example, using the product and its API with a REPL tool and SQL to wrangle and evaluate data. In general, users tend not to interact with the product in an instrumented way, but testers might do so quite profitably.

In most people's minds, so-called "manual testing" tends to be interactive, rather than unattended. Notetheless, testing can be experiential without being interactive from one moment to the next; for instance, when stepping away from the system and letting the machinery handle a large processing job, just as some contemplated user might.

Experiential testing work, whether interactive or unattended, exploratory or scripted, involves all of the usual information sources and oracles and tools that the tester normally uses and applies. Experiencing the product is only one element of what people unhelpfully call "manual testing", yet "experiential testing" expresses that aspect of what testers actually do far better than the former term. For more: https://www.developsense.com/blog/2021/08/alternatives-to-manual-testing-experiential-attended-exploratory/

Without experiencing the product as real people do, our testing will suffer from vitamin deficiencies and malnutrition. For the tester, experiential testing extends the range of the tester’s learnings, strategies, models, ideas—and feelings. All of these can point to problems in the product.

We save time and money for the organization when we experience those problems on behalf of others—before it's too late.

If you've got this far, a tip of the hat and thank you. Rapid Software Testing Explored Online for Europe/UK/India runs June 27-30. It ain't just for testers! Developers, managers, designers, support folk, etc. are welcome too. A class description is available here.

James Bach will be teaching RST Applied June 6-8; and RST Managed June 20-23; all for North American time zones. Visit his site for specifics.

The entire public schedule for Rapid Software Testing, including classes from me and from James, is available here.