Testing Alone and Under Fire
When we are alone and cut off from professional advice — and many of us work in situations where we are the only QA on the ground, or work in organizations without experience, methods, tools or process — can we formulate a general set of rules to help us survive and test well?
In Young Men and Fire, Norman Maclean tells the story of a group of fire fighters battling a brutal wildfire which burned above the Missouri river in 1940s Montana. Once dropped from a low flying airplane in to the disaster zone, they were alone and cut off from communications, able to trust only in their tools and themselves. At the end of a long day, 13 people were dead on a hillside.
This prompted the USDA Forest Service to develop a set of 10 iron rules for exactly this situation which they called the Standard Firefighting Orders. These rules were modeled on an earlier, similar set of rules for US Armed Forces sentries, which often need to deal with situations without being able to call for precise orders first.
In Software Testing, people are seldom dying because of our activities (although not always, see When Software Kills). We are, however, constantly in the business of trying to detect fires and fight them before they become serious. After all, this is why we have smoke testing ;)
But like firefighters on a smoking hillside, the question we may need to ask ourselves is, when we are alone and cut off from professional advice — and many of us work in situations where we are the only QA on the ground, or work in organizations without experience, methods, tools or process — can we formulate a general set of rules to help us survive and test well?
While acknowledging that each testing situation is unique, let’s give that a shot. Following is my take on a set of “standard testing orders” based my experience of surviving hard situations and some first principles (and feel free to add your own or take mine down in the comments below).
Standard Testing Orders
Here they are, in order:
1. Prioritize activities that find bugs
2. Make sure developers test their own code
3. Make sure bugs get fixed
4. Ask how this is going to break
5. Ask how this is going get used
6. Have an approach plan
7. Test small things early
8. Test large things late
9. Have an escape plan
10. Allow for feedback from the field
Now let’s look at the reasoning behind them.
1. Prioritize activities that find bugs
In one of my previous jobs, I remember a team with a dedicated tester who worked for an entire year without finding a single bug that needed to be fixed. That tester made sure that the system worked, but as value-for-money it was a poor deal. The loaded cost for a tester on payroll was probably around $80K/year. This team could have saved that money and effort, and the customer would have received the same result.
Finding just a few important bugs, however, changes that calculation completely. In other words, what makes your job worth doing is finding bugs. Hands down. First and foremost.
It’s not the time you spend, or the number of tests you design or run. It’s the value of the bugs you find that leads to better software. So prioritize all actions that help you do that as fast and as well as you can.
2. / 3. Make sure developers test their own code, and bugs get fixed
A few years ago, I consulted on a multi-year project where management made the dubious decision to tell developers not to unit-test their code to save on delivery time. This project then spent a year in system test, overran cost by tens of millions, and logged a spectacular 10000+ (in words: more than ten thousand) defects.
Whatever you need to do to get developers to review and test their own code, it will make your life easier. Much easier. You can avoid entire categories of bugs which are easily testable in the code, like error handling, data type issues or boundary value problems. And if you lint and scan your code, you can get early insight into code quality, security vulnerabilities and performance problems (e.g. unbounded loops).
Granted, your existing culture, or indeed your systems (see above), may not fully support this. But an important aspect you can influence is having a trusted relationship between Dev and Test. This means that you should work closely together, and that the feedback you provide is worth paying attention to and that found issues are being fixed.
This mutual respect and productivity is greatly aided by speaking a common language. This means that you need to be able to understand system design and code well enough to talk about where and how to most effectively test them. It is equally critical to reach consensus on how products are going to get used and what is expected of them.
And this can really only be done by having a culture of communication and by asking about things. Which leads me to my next point.
4. & 5. Ask how this is going to break, and how this is going to get used
The first burning hillside I was ever dropped on was a job where I needed to lead a 25-strong offshore test team for a Fortune 100 client who was about to cancel their contract. The client was unhappy because the test team had designed 20.000 (in words: twenty thousand) test cases based on documentation only, without ever talking to either the client or the developers coding the solution. The way we won the client back was by scrapping the approach completely, and instead implementing a tight feedback cycle on every use case and piece of code.
It’s easy to forget that testing is a risk-mitigating activity. And that the easiest way to understand what is at risk of breaking is to simply ask.
Developers generally know where the system and the code are solid, and where they are vulnerable or their quality is unknown. Ask them how the system is going to break, or what worries them. Then prioritize your testing on possible problem zones.
Similarly, your product experts will know how the software is going to be used. They will also know what is critical, and what is less important. Ask about that, and make sure developers and testers understand that context so that you are building and testing the right solution in the right order.
6. Have an approach plan
When you ask and care about product needs and technical concerns, you also have an opportunity to have the reverse conversation about what you need for testing.
This is where you develop your approach plan. The thing about this is that it elevates your game. It’s the difference between being able to work on something small (a story, a sprint), and being able to handle testing of any size.
A good approach identifies the drivers for testing (what is going to break, how it’s going to be used, what people care about, what constrains you), and then makes testing choices that best meet those drivers.
In simple terms, if we’re worried about a technology we have never used or built before, we plan to test it well and early. If we know we have a must-have feature, we’ll plan to test that. If we have limited time, we prioritize by risk and test broad and shallow to smoke out problems. And so on.
7. & 8. Test small things early, and large things late
If there is a general rule of thumb about avoiding big fires, it is to fight them when they are small.
Most software is assembled from smaller components and then integrated into larger applications and systems. To keep quality problems small, try to test components as they are built out from small to large, and test things in isolation before integrating them.
One of the advantages is that you follow development very closely and provide immediate feedback on delivered code, which builds trust and saves time (in other words, you don’t wait for them to burn the toast before you scrape it). Another pro is that you can root out different classes of bugs at different scales, finding each at the earliest opportunity and progressively de-risking the system. This is known as the testing pyramid.
For example, when testing code units and small functional flows, you can find most simple code problems with I/O validations, boundaries, processing logic, or error handling. When adding mocked or real integrations, you can start finding problems with schema mismatches, queries, and data flows. When adding user interfaces, you can start finding obstacles for use and round-tripping. And when assembling full systems, you can start looking for problems that stem from system complexity, end-to-end functionality and data flows, and from running on target hardware and under production conditions.
(This, by the way, applies to non-functional testing in the same way that it applies to functional testing.)
9. Have an escape plan
One of my early experiences in dealing with legacy systems (think mainframes, green-screens, and robot arms moving tapes around) was to learn how easily they break. On one particular system, legend had it that a developer chose to correct a single comma in the master branch on the evening of a big launch without telling anyone, and the next day, six months of careful work blew up in production.
The lesson here is that the ultimate test is releasing to the wild, and that despite your best efforts unexpected things may happen there. So similar to having an approach plan, you should have an exit plan.
Some good strategies here include:
- Keeping both old and new alive and compatible, and switching from one to the other (risks are kept small by deploying dark or behind feature flags, or by keeping changes backwards-compatible)
- Having a roll-back procedure
- Deciding to allow problems to happen and to ‘fix forward’ (i.e. in production or on a short ramp to it)
However, knowing what to do depends on knowing when bad things happen, which leads to the final general order.
10. Allow for feedback from the field
If releasing to the wild is the ultimate test, this, of course, means that you need ways of learning what happens out there.
There are generally two feedback loops you should have, one for the technical parameters of how your software holds up, and a second for the user experience.
The technical aspect is covered by monitoring systems, alerts, and logs. They will tell you if servers are up and traffic is served, and will help you trace problems when they happen.
The user, on the other hand, generally needs ways of reporting problems (i.e. technical support, submitting issue reports) and reacting to the value of features in situ (such as in-app contextual support and feedback gathering).
Both of these feedback loops are critical in the modern way of delivering software the agile way and with a DevOps mentality.
That’s it. 10 general orders. 10 good things to do when you find yourself alone on that hillside. Hope it helps you turn fires into fireworks!
