Stop trying to measure test coverage!
A couple of weeks ago, I saw a guy asking on Twitter: Does it make sense to have 100% test coverage in a front-end project? After reading some comments, I stepped up into the conversation (because it is Twitter, that is what we do there) and asked him back: What does test coverage mean to you? But I never got an answer.
I usually see people interpreting test coverage as three different things:
- The number of lines of code executed by tests within a project.
- “How much” of your system is tested.
- The % of cases from a test suite performed before releasing.
So for the first one, I will call it “code coverage”. It is a well-known measure that may indicate how tested your system has been. Anyway, there is no magic number for it. Even if you cover 90% of your code with tests, the more relevant scenarios may stay under the uncovered part. The same goes for higher numbers: even with 100% coverage, it’s possible to have weak assertions (i.e. tests that only goes through lines but don’t check what is needed).
Make sure your tests are well written and not just “covering” lines of code.
For the other two points, it is hard to state. How do you measure “how much” tested is a system?
I would say that there’s no way to measure it. The testing process involves taking actions, learning about the system, and comparing the obtained results with the expected ones. It is a continuous process that never ends.
What do I mean by saying that it’s endless? A basic scenario: you are testing a user interface that has many input fields, buttons, tables, checkboxes, actions, and tabs, where you can do a lot of combinations using all those elements. Every step you do has an expected result. You can combine all of them and repeat them many times to see what happens, and each combination may produce a different output.
So let’s take a simple sign-in page with a password recovery feature that offers a password change after the third try as an example:
- Successful sign-in;
- Unsuccessful sign-in with non-existent username;
- Unsuccessful sign-in with invalid password;
- Three unsuccessful sign-ins in a row;
- Success after resetting password;
- Unsuccessful sign-in after resetting password;
- Reset password;
- Try again after resetting the password;
- Ten unsuccessful sign-ins in a row;
- Reset password twice;
- Reset password using the same old password;
- …
Do you see what I’m doing here? And I’m not even listing input things like charsets, sizes, formatting, injections, whitespace, and so on. The point is: every single action can happen countless times, and they can be combined to produce, or try to produce, a different output.
It is possible to create test scenarios for applications all day long. Of course, you have to stop somehow by considering some constraints like time, priority, risk, etc.
Then again, how can you measure the % of something that you can’t count? You can build test suites and say: “I test 100% of my test cases”, it doesn’t mean that bugs are not there. How good are your test cases? What are they covering?
Bugs will be always there! And that’s fine! You need to take care and try to make sure that the worst ones are not going live.
So please, stop trying to find mystical numbers to measure the quality of your applications. It is a lot more about what you are testing than how many tests you run.
Numbers are cool, they provide a way to measure things and achieve goals, but it’s hard to measure quality!
More than achieving numbers, quality is about:
- Your applications are doing the right thing, meeting your customer needs and solving the intended problem;
- You are doing it right, enabling your customers to go smoothly through their “happy paths”.
