Getting ready for test automation with better testability

During a chat about various topics, Mark Winteringham asked me ‘what is the impact of testability on what and where you can automate?’

First things first, the main means of increasing testability is to do some testing. You can assess code and whole systems for testability but there is no substitute for the real thing. Hard to test code, try and add a unit test. Hard to test system, attempt to do some exploratory testing that can stay in the vicinty of the target for 10 minutes. Testability is generally poor because there is no feedback (and no one looking), so get your testing effort started.

Happily, attempting to add lower level tests and getting stuck into exploratory testing is a great way to get your automation approach going. Trying to automate acceptance level tests (invoking large, connected parts of the system) as a starting point can be terrible. You end up automating what the system does (as opposed to testing for what it needs to do) and baking in all the weird behaviours of your application. Its those weird behaviours that decrease your testability, generating bugs, confusion and automated tests that fail for reasons and get ignored.

Sometimes you have to resist the urge to commit to acceptance test automation. Build some tools, learn your test data, fix some problems and do some exploratory testing. But when are you really ready?

Readiness

Sometimes your application isn’t ready for acceptance test automation. Its too unstable and you just know that you’ll spend more time diagnosing failures than finding regression issues. Remember, a key indicator of testability readiness is:

'Can I sustain a 10 minute exploratory testing session which stays in the area set by the charter?'

If every session ends up chasing weirdness that is way off charter, its a good sign your system isn’t ready.

Although sometimes its worth it, depending on the context.

Here are two recent examples:

1. Hard to test, needs acceptance test automation – I was helping out a company with suite of web applications. What had been built had been around for a long time and testing was an afterthought. Testability was low, having acceptance test automation was going to require commitment and maintence. However, some of their web applications had been down (returning an error due to an unset environment variable) in test and production for weeks without anyone knowing. In this case, adding smoke tests despite poor testability is really worth it, as the alternative is much worse. The tests were not as resilient as they could be but it was worth it. We could then invest more in testability to enhance these tests and the applications themselves.
2. Hard to test, hold off on acceptance test automation – in this example, I was testing a mobile application using an on device datastore which used a third party library to synchronise data to remote storage. Data synchronisation could happen now, in a little while, later on, next time you change the app lifecycle, when the data changed on the remote storage but only in certain contexts. Controllability was poor, thus acceptance test automation would be difficult. We actually tried it and it was bad. There would need to be significant investment into hooks to clear state, synchronise, check remote availability to name a few to add the required controllability. In the end, it is better to let the lower level tests (unit and widget tests in this Flutter example) rather than creating another problem for ourselves. Some technologies just resist attempts to test them at higher levels, such is the price of your choices.

Two examples, both with hard to test but acceptance test automation was needed to solve a problem in the first case. Judgement is always required, as in some contexts, adding the tests can drive improvements in stability. Testing once again increases testability.

Human and Machine Readiness

I think its true that you can have a system which has excellent testability for machines but less so for humans and vice versa. Alan Richardson describes this well with regard to logging:

• A useful log for a human is something that describes internal state, that can be read as a narrative. Many entries over time.
• A useful log for a machine is something that is enumerated and easily parsable. Usually waiting for an event to happen.

As usual its slightly more complex. You can add a set of thoughtful identifiers in the DOM of your web application which are great for automation and great for making your application more accessible. If you are going to make your application more testable to get it ready for automation, try and focus on these multipliers. Testability can also be measured by the variety of testing types you can meaningfully do. If you are not doing security and accessibility testing, it can be a testability as well as a skill/will/time issue. If you add anything to enhance testability, think of how useful it is for a human and how useful it is for another machine to interact with.

Finish

To finish, some readiness patterns to keep in mind for test automation:

• Low testability can confine you to lower level tests, as invoking the whole system is just too unstable.
• You are confined to the lowest and highest environments (local and production).
• You can only run load tests in production.
• You find yourselves mocking things you shouldn’t to get the tests to pass, until there isn’t much meaning in your tests left.

For all the words, there is no substitute for seeing exploratory testing and test automation for what they are, not different but part of testing a thing well. For me, Maaret Pyhäjärvi’s approach to contemporary exploratory testing describes it admirably. The testability gains from approaches like this could be enormous for your team.