Why I prefer a Test Contract to a Test Plan

Some context…

I firmly believe that quality really is everyone’s responsibility. In order for that to become something practical and useful, the things we do have to be easily accessible to everyone, regardless of their background. Sometimes that means we have to simplify and abstract away the actual complexities of practices and techniques, particularly around testing. This can be troubling for some people, understandably so.

Here I’ll go straight in to what a Test Contract is, how to create one and how it’s used. If you’re interested in the thinking/philosophical side, scroll down

In Practice, what is a Test Contract?

In its most base form, it’s an agreement on how and when pertinent risks associated with a project/feature/thing are going to be managed.

It’s made up of:

1. A synopsis of the project/feature/behaviour/change/thing
2. A list of Quality Attributes decided for/by the business
3. A set of risks that need managing for each Quality Attribute
4. A decision on when each risk should be managed
5. Ideas on how to manage the risks

This is what the current template looks like

In this made up example, the team are adding the capability to pay for part or all of an online basket, with a gift voucher.

The team will have recently discussed the feature to an appropriate level of depth. In my organisation, we use a version of example mapping as a group to describe the desired behaviours from multiple perspectives/personas.

When there’s a good understanding of what behaviours are being built, the team, using both context derived and agnostic heuristics for reference, will consider potential risks to each quality attribute. Each risk is ideally phrased as an open question. Every risk conceived by the team, will be added to the test contract.

As each risk is added, the team agrees on when it’s best to manage that risk, using 3 levels

• Level 1: It is a risk, but will be managed by a practice/standard/convention already in place (green sticky)
• Level 2: It is not yet being managed by anything specific, but we can manage it effectively as we build the behaviour i.e. through TDD/good design practices/exploratory testing sessions on small increments (orange sticky)
• Level 3: Whilst we can make best endeavours, we accept that we won’t be able to properly manage the risk until the feature has been in the wild (red sticky)

Finally the team will share ideas on what activities might be useful to help manage the risks. These get added to the High Level Approach section.

And that’s how a Test Contract is created.

How are Test Contracts actually used?

Whilst there’s an initial discussion to kick things off, the real benefit of a Test Contract comes from referencing it continuously during development. Because it’s so easy to understand and get value from, it’s quick and easy to reference during any sort of conversation. Also, it’s just as easy to update.

1. We’ve managed a risk through TDD? Great, let’s change its colour from Orange to Green.
2. We’ve found an emergent risk as we’ve discovered more unknowns? Great, let’s add the sticky to the contract and update any High Level Approach we need to.

One way we’ve done this is to keep the contract in Miro with all of the other collaborative practices we take part in. That way it’s always easy to find. There’s no searching through Notion/Confluence or Jira/Shortcut. Even better, it’s there when we explore together. These questions are used to frame an exploration goals and help to get things moving. No more – “what should we test?” answerless questions, starting becomes easy.

What do I think a Test Plan is?

There are many forms of Test Plans. When I think of them, I first refer to the IEEE 829 standard, an example of which can be found at https://www.ecs.csun.edu/~rlingard/comp480/TestPlanTemplate.pdf

Thankfully there are many optimised versions and a good example of this, in my opinion, is the One Page Test Plan by Claire Reckless https://www.ministryoftesting.com/dojo/lessons/the-one-page-test-plan

In principle, what is a Test Contract?

The most important thing about the Test Contract is the name. I’ve picked the word Contract specifically because:

1. It implies an agreement between 2 or more parties
2. It implies that there’s a level of commitment to adhering to the content

This sets the principles for the success of any Test Contract

1. It has to be created, negotiated and agreed to by those it directly impacts
2. It has to be acted upon in good faith

This implies that the Test Contract is not written by

1. A ‘tester’
2. A group of ‘testers’
3. Any other individual
4. Anyone who will not feel the impact of the content

What are Quality Attributes?

To make the practice of building and working with Test Contracts inclusive, I believe that they should be driven by a set of Quality Attributes that the business believes are most important to them. This is a whole post in itself, however in short:

1. A Quality Attribute is something about the software that defines its success within the domain/market it operates
2. The Quality Attribute matters to the point that, if something about the software threatened that attribute, it would be actioned/managed/rectified and not ignored
3. Everyone should know and understand why they’re important for the business

Examples of Quality Attributes that might be ubiquitous

1. Easy to use
2. Secure
3. Inclusive

Ideally however your Quality Attributes will be more specific to your particular domain.

One thing to consider. Whilst these sound a bit like ‘non functional requirements’, and they can be useful to add to the discussion, don’t be tempted to grab a list of non functional testing types and use them directly. It’s much more useful to describe things in a way that adds context to what you’re trying to achieve.