I admire the simplicity of TDD.
Red, Green, Refactor.
Write a test, see it fail. Write some code, see the test pass. Refactor your code, the test should still be green.
It’s simple to describe and a simple concept to understand. Except it’s not simple. I rarely practice TDD by myself these days, but I clearly remember how hard it was first picking it up. It took a lot of frustration and failure to get to a point where I felt comfortable.
Compare that to Testing. Testing is hard, just like TDD. Testing also feels hard to understand. There’s Strategies, Plans, Frameworks, not to mention the nebulous nature of knowing what and what not to test. You might’ve worked with a great software tester. As with all experts, great practitioners will make things look and feel simple, but they will all tell you that it’s not.
For a long time I’ve been thinking about how I could simplify my testing world. To make it easier for me to hone my own craft and if possible, help those I work with get over that initial barrier. This is where I’ve got to.
Explore, Act, Improve.
Explore ‘something’ to gain knowledge. Use the knowledge to act. Improve how we explore.
The current working title for this process is Action Driver Exploration. Better suggestions are very welcome!
Déjà vu
As with all things, nothing is truly new. As I’ve gone through my career I’ve reached points I might classify as step-changes in my thinking. At the time I might have felt I’d reached each naturally. However I’d soon discover that at least one other person had come to a similar conclusion before me and described it.
Unsurprisingly, the same thing has happened here. Thanks to the wonderful Mike Harris I was introduced to William Edwards Deming. Deming described a Continuous Improvement Cycle. The cycle goes: Plan, Do, Study, Act (PDSA).
I’ve semi-joked for years that more appropriate title for what I do is Software Scientist. Deming references the Scientific Method in his work. There is no coincidence here. The Scientific Method underlines PDSA. In practicing PDSA we build a scientific theory of the business. We then test and inform that theory with experimentation, building further knowledge of how the business functions.
Deming’s scope for PDSA is to improve the business. I’m interested in the business from the perspective of the software it builds and services it offers. There’s a lot of overlap to explore. I’m still learning about Deming and PDSA and I’m sure that over time his work will influence mine further.
Explore, Act, Improve
The goal of EAI is to create a feedback loop for our testing practice. It aims to balance the ratio of test activities and knowledge gained, with the quality of the actions taken. I believe we can do this by studying the intersection between what we discover about the software we build and the business perception of how useful that behaviour is.
I believe that the quality of our testing is determined by the quality of the actions our testing informs. Each loop of the EAI cycle is designed to help us better understand how the software behaves and how that behaviour impacts the business.
If we trended this to infinity, all testing activities will be followed by high quality, impactful actions and waste will be removed. This is of course impossible and should not be set as a goal for implementing this practice.
Step 1 – Explore
This first step encompasses all that you might currently think of as testing. It includes the strategising, analysis, planning and execution of testing on ‘something’. That ‘something’ can be a feature, the whole system, an idea, a design, UI wireframes, deploying your software, the analytics and logs for the last few days of customer use. Anything we believe we would get some value from, if we explored and studied it.
From this point onwards, I’m going to replace the word ‘testing’ with exploring.
If you’re new to exploring, my recommendation is to just start. Use whatever knowledge and experience you have to do the best exploring you can. It’s better to start than to over think. Once you start, the feedback loop can begin.
The most important part of exploring is gathering useful information that describes what you discovered during your exploration. Diarise everything. Take screenshots and debug/console/logging information after each ‘action’. Record how you felt and the thinking process that resulted in you doing the next action. Consider making predictions and jotting them down before taking the action. This practice is particularly useful because it helps build a habit of asking questions. Having a question beforehand is a good guide. Exploring an hypothesis can make things easier, but it can also put the blinkers on. Learning when to step in to something specific and step out to take a broader view of the landscape, is a skill in itself.
Ready to go exploring? I recommend starting with the following:
- Pick a short period to time box your exploration. 20 to 30 minutes maximum
- If possible, explore with others. It’s more interesting, you can share the diarising part and you’ll each bring different perspectives, sparking conversation
Starting to explore is that simple. If you’re after something a little more detailed, I recommend this excellent overview of Exploratory Testing by Maaret Pyhäjärvi.
To help make my exploration more effective, I’ll feed in to my exploration Quality Strategy for. For me a Quality Strategy is tiny, but the information it contains guides me to create useful questions. Useful questions are more likely to result in the discovery of information that drives impactful actions.
It contains:
- The Mission – The reason why the software exists
- Quality Attributes – A list of 7 to 10 Quality Attributes that we believe are important to the success of the mission statement, to the point where if our software doesn’t meet an appropriate level in one or more of those attributes, we would take action to rectify it
- Team Values – A list of 3 to 5 Values that tell us something about what we believe in and matter so much to us that, if we observed a deviation from those values, we’d put in corrective measures
I’ll cover this in more detail at a later date.
Step 2 – Act
We’ve been exploring and we discovered a number of interesting aspects about the software we’re designing, building and operating.
In the past I found it easy to move on to something new once my discoveries had been passed on to others to “do their thing”. There’s always more exploring to do. My feedback will have been submitted, discussions happened and bugs added to the database. Next habit and demand kicks in. Another ticket needs my time. More exploration must happen.
The likelihood of this outcome, of moving straight on to more exploring, often correlated strongly with certain types of project and culture. The less ‘Agile’, the more likely I was to ‘complete’ a testing activity and move on to the next. If I look back and assess this based on my ideals today, my behaviour resulted in less impactful exploring.
Why is that?
Example 1
How many times have you the reader submitted a bug, only to find that it was ignored? Maybe it was continuously deferred down the priority list, or closed as ‘will not fix’?
Example 2
When writing an automated test, a new idea comes to mind. It’s relatively simple to author it, so you do so. It gets included in CI/CD pipeline. After a few releases it fails. It’s not flakey, the test has found a genuine problem, but the release goes ahead anyway. The test fails during each of the next handful of releases, but with the same outcome, the release goes ahead.
If we could model and define our exploring activities perfectly, all exploration would result in an impactful action on the product, software, service and/or business. For me, and you might recognise this, the most satisfying impactful action from my exploration is a change made to the software we’re working on. If I find a bug* and the software is improved because of my discovery, I feel like I’ve contributed. I feel that my time and effort was worthwhile.
What I didn’t realise back then was, I wasn’t always prioritising my exploration for impactful actions. Sometimes I’d explore because I was interested. Sometimes because I thought it was important. Sometimes I was just doing what I was told. Looking back at this now, across a distribution of reasons why I explored, there were many instances where my exploration resulted in no meaningful, impactful action being taken. My desires and ideas were not fully motivated by what the business wanted and needed.
A note of caution. The business does not always know what it wants, nor what is important. As a tester it’s important for me to remember this. It reminds me that exploration is inherently lossy. I will have to take risks, just like a racing driver does as they push their car to its limits, to achieve the best lap time. Sometimes that results in a crash. Our exploration is unlikely to be as catastrophic.
So this is not an exact science?
No unfortunately it’s not. However, if I could go back and coach myself, I would absolutely encourage the younger me to focus my time on exploration that had a higher probability of an impactful action.
I want my exploration to be and be perceived as, impactful.
Here’s a list of impactful actions that I’ve experienced gratification from
- The customer experience was improved by fixing a bug
- The design was simplified by removing code
- The architecture became more secure by removing attack vectors
- The pipeline was streamlined by removing tests that wouldn’t stop the release, if they failed
- The cost of running the service reduced by shutting down unused environments
- The conversion rate went up by reducing the response time of a related API
- The Support team had fewer issues to deal with because an error message was made more clear
These and more all happened because of good exploration. Not just of the software, but because the exploring activity found an intersection between the software and the business, that mattered.
Even when this doesn’t happen, we have an opportunity.
* Loose definition of the term bug. To mean anything which wasn’t intended or could be made better.
Step 3 – Improve
Which leads us to the final part of the loop – Improve.
If I focus the assessment of my exploring on impactful actions, it enables me to do two things
- It makes it easier to recognise when I’ve done good exploring
- It gives me a concise measure to help focus the analysis of my exploration, helping improve it for next time
You may have noticed it already, but there’s a paradox here. “Good” exploration results in action. “Not so good” exploring results in no action.
However, if I can do something to improve my exploring, that must be seen as an impactful action, right? If I can increase the probability that future exploration will result in actions that directly impact the product and business in a positive way, I’ve done some useful.
But beware! If the only action that comes from our exploration is improving future exploring, then it’s unlikely we’re improving it at all.
Practice does not make perfect. Only perfect practice makes perfect.
Vince Lombardi
We need to make sure that our learning is also relevant and impactful. We can do this in a similar way to recognising useful tests in a CI/CD pipeline. Referring back to Example 2 above. If a test fails in a CI/CD pipeline but we ship anyway, did that test provide any value? In one sense it did not. But if we look at it with a growth mindset, it can suggest we’ve learnt something new about what the business finds important.
One interpretation could be: the risk of a feature not working is easier to deal with than not releasing this iteration.
Another way to interpret this situation could be: the pressure on an individual or the team is sufficiently high enough to risk quality in order to achieve an objective.
Using this loop I can make a more informed choice on how to behave. My observation of previous decisions and their outcomes can help me to determine whether I needed to explore this further. Is it an OK risk for the team to take? Or do I feel there’s a better decision to make, based on my study of how similar decisions have turned out previously? My gut might tell me one thing, but my knowledge could contradict it. Sharing feelings openly and authentically is a good way to start a conversation. Backing up those feelings with data will make that conversation more productive and increase the chance a more informed decision will be made. I can only do this if I’ve been following this loop. By observing and collecting information about the software and the decisions made based on that software.
Referring back to earlier, this learning loop helps me identify the intersection between the software and the business, where exploring will more likely result in an impactful action. This requires context and an ability to perceive a wide range of information and input. This is difficult! To help me with this problem, I devised a model which I call Context Derived Heuristics.
Context Derived Heuristics
Heuristics are great tools to help us make decisions. If used appropriately they will increase the chance of a good outcome.
In my experience, most heuristics are described at an abstract level. This enables them to be applied across multiple contexts. You’ve probably come across DRY (do not repeat yourself).
There are more detailed heuristics, such as this Heuristic Cheat Sheet from Elisabeth Hendrickson.
A Context Derived Heuristic is different because
- It was discovered within the context of the problems we’re trying to solve
- Is described in a way that is not designed to be used outside of the current context
- It has a specific format
At its most basic, a CDH contains
- Context
- Action (rephrase)
- Impact
- Analysis
These are crude examples, but they do give an idea of how I structure them. I recommend keeping them in a digital notepad, allowing for easier organisation as your catalogue grows. I tend to organise them by perceived positive and negative impact and then by contextual area.
| Context | Action | Impact | Analysis |
|---|---|---|---|
| Entering credit card info in Checkout | Reducing network speed to 3G | Card info validation failed as the timeout was too aggressive | Showing unnecessary errors on Checkout is perceived to reduce customer confidence, negatively impacting conversion. |
| Add to Bag notification automated test | Failed in pipeline prior to release | The test found a bug, but wasn’t deemed necessary to fix prior to release. Will fix-forward at a later date. Removed check from pipeline run. | Passive notifications to customers are deemed not show stoppers. Option for optimisation in pipeline performance, if needed. |
I plan on delving deeper in to CDHs at a later date.
Completing the Loop
We’ve finished the first loop. Within that first loop we’ve explored ‘something’ within the context of building software. That exploration has hopefully resulted in impactful actions that have generated a recognisable feeling of satisfaction. Worst case, we’ve learned something about our exploration and created at least one Context Derived Heuristic.
Any CDH’s we have discovered and created can feed in to the next cycle of the loop. If it helps directly, fantastic! If it helps indirectly, by inspiring other ideas and ways to see the world, that’s just as good. Either way, we’ve begun the journey of formally understanding the intersection between the software we’re building and the how the business perceives that software. In traversing that landscape we’ll take steps forward, some steps back. By trusting the practice, we’ll gradually improve our ability to explore the software and business with purpose. That purpose will lead to more impactful actions. Our understanding of the Mission and the Quality Attributes that really matter, will deepen. We’ll also better understand what our real team values are. Those that stand the test of time, pressure and reality.
This has been a very high level overview of my exploration of how I believe I solve testing problems and improve the effectiveness of my testing over time. Even writing this has challenged some of those beliefs and uncovered further insights, leading to minor changes to my perception.
Even so, it’s a lot to take in.
It feels to me that Action Driven Exploration shares the same “simple to pick up” attribute that TDD has. It also shares the same “devil is in the detail” attribute. It’s a practice. One that will take time to master and see its full benefits.
It’s taken:
- The whole of my career so far, for this practice to evolve
- The best part of a year for me to be able to recognise the practice as something tangible
- Four weeks to describe it at this basic level
I’m really excited to explore this further, to describe it more deeply and share how I use it. Especially now that it’s something that’s not just an undescribed habit, living in my head.
Addendum – A Journey
It’s taken me nearly 20yrs to come to this point.
Despite many different companies, products and challenges, how I test software hasn’t changed too much.
What really has changed is my articulation of my experiences of the craft.
This is a crude history of my journey.
- In the early days I recognised I was good at knowing ‘what to test’ and where to find interesting information. I would explore and identify problems. On the whole, those problems got fixed.
- As my involvement in building complex software systems grew, I recognised that problems discovered and fixed were effective inputs to the feedback loop for automation coverage.
- Years passed and I became more influenced by others in the testing community. This helped me discover problems found by my testing could be better articulated as ‘knowledge of how the software behaves’. This knowledge came with a power. Describing how the software behaved challenged the assumptions of how we expected it to behave.
- I’d encouraged developers and new testers to focus on the skill of presenting their knowledge, but not necessarily to do so in order to increase the likelihood of action. If I test, gain knowledge and present it, but no one cares about it, no actions are taken. Whilst this didn’t happen all that often, others in the community have highlighted that knowledge alone isn’t enough. More recently I’ve become better at recognising that knowledge can be useless without explicit action.
- Providing testing with a high action coefficient became a priority. More actions led to faster iterations. Faster iterations led to a reduction in learning. A reduction in learning, with leaner and leaner automation coverage, increased the likelihood of repeated mistakes, especially in hard/slow to automate areas of software. It’s at this point that I came up with the concept of Context Derived Heuristics.
Each notable change was years in the making. Years filled with daily practice, in real world situations. I was accumulating lived experience of the impact and outcomes of that practice. When I had enough experience, I was able to step back and critique that impact and those outcomes to assess what was missing. Influenced by those who’d been on the journey before me, I identified the next most obvious gap and tried to course correct. Those course corrections led me to now.
Thanks
Quite a few people have been kind enough to listen to me talk about this and offer their feedback and support. I adore every single one of you. Thank you for being there and helping me on this journey.
In no particular order, using their twitter handles…
Dragons for Elevenses 
2 thoughts on “Is it possible to improve the impact of our Testing by simplifying it?”