Test coverage: does 100% test coverage mean full assurance of quality?
Every product should have to deliver maximum value to users as a main goal. Value delivered and quality have a strong correlation. Therefore we can state an operation aims to maximize the quality of its product or service to deliver maximum value to its customer. This logic can be applied to any production, including software development.
Predictability and stability are two essential attributes when we talk about developing a new functionality. Users perceive predictability when, for instance, every time they press the same button, exactly the same event will occur. When the same event always occurs, in every scenario, they perceive stability.
Testing is a way to ensure the quality of some functionality. There are different types of tests and different metrics. Test coverage is one among many metrics that guide a team in reliable deliveries, it is related to unit testing. It measures the percentage of tested lines.
Test coverage = ( line of code covered by test * 100 ) / total lines of code
As application quality assurance is related to test coverage, could we state that 100% test coverage means full assurance of quality? Let’s elucidate and answer this question.
Arithmetic operations calculator
Let’s consider a calculator able to calculate the result of arithmetic operations through addition, subtraction, multiplication and division operators. Our calculator will only be able to calculate the result of an expression involving two numbers and an operator, not using parentheses.
We can architect this solution using the Calculator and Operators modules. The first is responsible for processing user input and the second for arithmetic operations.
Considering this architecture, let’s rephrase the initial question in terms closer to these elements: “If we have a module showing 100% test coverage, can we ensure predictability and stability in all cases?”
To answer this question, let’s focus the discussion on the Operators module.
Application setup
The project has been written using Python 3.8 and packages pytest and pytest-cov versions 7.4.4 and 4.10, respectively. Let’s check the list of files for this project.
- main.py
from src.calculator import operate
if __name__ == "__main__":
while True:
print("")
a = float(input("Insert value of A: "))
b = float(input("Insert value of B: "))
operation = input("Choose one operation: (+, -, *, /)")
result = operate(a, b, operation)
print(f"The result of {a} {operation} {b} is {result}")
answer_code = input("Do you want to repeat this process? (y/n)")
if answer_code == "n":
print("Thank you, bye!")
breakThis file is responsible for starting the calculator. Note that this only knows the Calculator module, as described in Figure 1.
- src/calculator.py
from src.operators import sum, sub, mult, division
def operate(a: float, b: float, operation: str):
if operation == "+":
return sum_(a, b)
elif operation == "-":
return sub(a, b)
elif operation == "*":
return mult(a, b)
elif operation == "/":
return division(a, b)The result of any operation is based on two numbers and a chosen arithmetic operation. A more complete solution would have treatment for an operation different from what was expected, but this is not our case.
- src/operators.py
def sum(a, b):
return a + b
def sub(a, b):
return a - b
def mult(a, b):
return a * b
def division(a, b):
return a / bThe functions in the Operators module are wrappers of operations that this programming language provides natively. Technically we could do without it, but the reason for its existence is its didactic purpose. We will answer the central question of this text necessarily using the arrangement described in Figure 1.
An important note is that we are overriding the built-in sum function. This is a non-recommended practice.
- tests/test_operators.py
from src.operators import sum, sub, mult, division
class TestOperators:
def test_sum(self):
assert 6.0 == sum(3.0, 3.0)
def test_sub(self):
assert 0.0 == sub(3.0, 3.0)
def test_mult(self):
assert 9.0 == mult(3.0, 3.0)
def test_div(self):
assert 1.0 == division(3.0, 3.0)This file has test cases written in it. Note that each operator could have more than one case. Let’s focus, however, on the fact that each function is tested at least once and the implication of this on the test coverage metric.
Figure 2 shows the final result of this setup. The venv directory contains files related to the virtual environment and the requirements.txt file contains the packages used in this project.
Operators module test coverage
Because they are basic arithmetic operations, I will dispense with explanations regarding which result should be produced on each operation.
python -m pytest --cov=src tests/Executing the command above at the root of the project produces the test coverage report for all modules.
Note that because there is no test case for the Calculator module. The values of the Stmts and Miss columns are the same, that is, no line of code of this module was tested and this implies 0% coverage. On the other hand, all lines of the Operators module are covered by test cases, then its test coverage is 100%.
100% test coverage = total quality guarantee ?
To keep the didactic approach, we will further isolate any factor that could confuse our thinking. Let’s assume that the user will only provide valid numbers and operations to the prompt. Otherwise, we would have to talk about exception handling and testing in the Calculator module.
That said, now let’s go!
Imagine a user using this application for hours, performing various arithmetic operations. Until, at a certain moment, the values A = 10, B = 0 and the division operator are inserted into the prompt. What will happen?
ZeroDivisionError: float division by zeroThis error message will appear. Although the Operators module has 100% test coverage, the case of dividing a number by zero was not considered. The consequence is this user will have unforeseen behavior and the application will not work in the same way as it worked for other divisions, compromising stability.
This error is easily solved by handling this exception. However, the point is that we have just seen an example where 100% test coverage did not fully guarantee the quality of the application.
If every effort to guarantee 100% coverage does not translate into full assurance of quality, what can we do to increase the quality level of our application? The answer lies in better test case planning.
Thinking on test cases
When we write test cases without criteria, we have no guarantee that the most relevant cases will be considered. This is exactly the case illustrated in this text.
A set of relevant cases should reflect the behavior that the tested module will have when used by users. Unfortunately, there is no universal rule for identifying them. However, test-driven development(TDD) helps a lot.
It is also useful to know the problem domain we are working on. Our arithmetic operations calculator, more specifically the Operators module, is in the domain of mathematics. Knowing that the division of two numbers with opposite signs results in a number with a negative sign comes from knowledge of the properties of this operator. And this could be a test case to consider.
Final words
Every time we hear that an application has achieved a certain percentage of test coverage, we must realize what that really means. According to what has been explained, it is not true that upon reaching 100% coverage we will have a total guarantee of quality. To have a real dimension of the quality delivered to users, we also have to consider the process used to write the test cases.
Another metric to examine is Code Coverage. Besides considering the number of instructions, it also evaluates the paths that the execution flow can follow. Conditional branches can provide multiple paths for a small piece of code. It doesn’t exactly solve the problems highlighted in the text. However, the use of different metrics can offer us a new point of view.
It’s worth remembering that there are other types of testing that must be considered, since unit tests don’t cover the entire application. Going further, the most effective way to move towards full assurance of quality is to foster a testing culture. This subjective factor is extremely relevant. The level of quality rises when the professionals involved in software development engage to effectively test what is being written.
Would you like to share an interesting case where testing significantly contributed to achieving success?
If you liked the text, consider sharing it with the community and give me some claps. This will certainly encourage the continuation of this work.
