I was recently talking with a group of people and I suggested to them that testing is not always the answer, to what many were bamboozled about.

Ok, now, don’t get me wrong, I work with quality and quality is my passion, I am not going to recommend against it, but there are some rough times in which there are other priorities, and I think it’s wrong to just advice to keep testing no matter what.

There are two main principles about tests and automation and, if you know me, you’ve probably already heard them:

1. Testing is about finding the right balance.

2. Automation is about saving time.

Both end up being relating to cost: for the first one, if testing something could be more expensive than going ahead without testing, then it is better to skip that test (or tests).

For the second one, if automating something is going to cost you more time than the overall added time that it would take you to test something manually, then you should not automate.

These principles might sound straightforward, but in reality, it is not always easy to make the decision to not test or not to automate something. It is against our instincts, “what if”s start popping up around us.

When should we stop testing?

I remember this was a question I was asked during my interview process back when I joined Microsoft. “When should we stop testing?” I gave my honest answer back at the time, which I don’t remember fully because, well, I tend to get very nervous during interviews. (If I’ve ever interviewed you and you felt you didn’t do well, believe me, I’m not going to judge you on that, we all have a bad day)

I think to remember that I replied something on the likes of: we never do. More than 10 years after that, I feel more confident to give an answer: we stop testing when testing is more expensive than no testing.

You need to be able to have a clear understanding of the risks of having issues around a system or feature in order to make the decision of not to test it. Basically, you need to be able to address all those “what if”s that were popping up before.

In case of doubt, or if you have enough time, test as much as possible, do not ever stop testing, but if testing is costing you too much,  maybe consider stopping. For example, if we might be late for a promised release and we can lose uses if we are.

This particular scenario is really not a situation one would like to be in, and if it happens it might indicate poor planning, features not being properly set or testing happening too late. However, in that sort of situation, you need to “pick your poison”.

The most likely and preferred case is that you have a good amount of tests, although you could always create more. Extensive tests might also end up taking too long to execute.

The neverending test suites

There are several possibilities for a company or team to have too many test cases. It could be a case of lack of trust in the product or other team members, so things are tested more than once.

I’ve also seen companies that would praise or compensate somehow their employees based on the number of test cases they write or automate, as their quality or effectiveness is much harder to measure.

When test suites start expanding, their maintenance gets more difficult. If documentation is not good, it would be increasingly tough to understand what test cases make sense to run against what deployments. When something fails, we need to decipher if it was an actual failure, something needed to be maintained on a test or something related to some external dependency.

It is ok to have a long list of tests, as long as they have the following qualities:

• Each test tests a single thing
• They are well documented
• They always produce the same results under the same circumstances – this is called ‘idempotency’
• They can run in parallel
• They can be evenly grouped by feature or priority
• They don’t have dependencies unless they are testing a dependency

I call these SWIPED tests.

Besides this, you can also have a list of BVT (build verification tests), that you can run as part of the regression. The ideal tests for this have the following characteristics:

• They test a high priority, a high risk or a core functionality

Finding balance across the test pyramid

Another point in which we tend to increase the complexity of our quality phases is when we have no trust in the test pyramid. This happens typically when different teams are working on different levels of the test pyramid.

For example: if I am not sure if the level below me has checked everything, I will check it in my level. If I ever caught an issue they missed, then the trust is fully broken and I will keep testing as much as I can, including things that might have already been covered at a different level.

However, the higher up you go on the test pyramid, the more expensive the tests are, the longer they take to run, the less possible it is to maintain them and so on.

Therefore, it is important that we test the right things at the right time. Nor more, nor less. That is the principle of the test pyramid and  why it exists to begin with.

What should we not automate

Last, but not least, let us look a bit into automation. If you know me at all, you’ll know that automating is crucial for me. Avoiding repetitive tasks saves you time, which is your most valuable asset.

I think that everything could potentially be automated, but the objective here is to save time.

There is a scenario, though, in which it is OK to “waste time” even if you do not get to finalize an automation. That scenario is when you are researching. It is ok only because when you are researching, you are learning, so that is not a wasted time after all. According to Malcom Gladwell in the book outliers, many successful people became successful because they could afford failing many times before their success.

Note here the word “afford”. We are again talking about costs and savings. In this case,
in an even more literal meaning of the word.

Many managers would not be very happy to hear this, but failing is part of the process. I believe we should allocate some time to research as part of our feature plans. Then again, do this only if you can afford to invest time in research.

Personally, I think we should try to allocate some time to research and to try new things in our spare time as well, whether that is oriented to our careers or to something else we want to learn.

As we mentioned before, careful planning could save you from ever being in a position that you cannot afford to continue testing, as we should never promise things that we would not be able to deliver. For that, it’s crucial that all the stakeholders of the system under construction are aligned, but that’s well..another story.