Test Coverage
Test Coverage in QA is a very confusing topic. The main question is: how do you measure it? It is a subjective assessment of what percentage of functionality is covered by existing test cases. There is no standard methodology to calculate it. If you ask 100 different testers about the same suite of tests, you will probably get 100 different answers. It is hard to accurately track changes in this type of test coverage and make valuable decisions for further testing. So why do we need test coverage? Well, it’s important to know whether you tested all the functionality and didn’t miss anything, so you can put your QA stamp on a release.
Code coverage, on the other hand, is a straightforward metric. It is a Percentage of source code executed while tests are running. (and in the traditional definition it’s specifically unit tests ). This measurement is accurate, the metrics are automated and clear. Unit tests run during build time, and a code coverage report is generated. Many teams have a certain target set for their code coverage, i.e. no less than 80% and they try to never go below it and ideally be close to 100%. I believe Google is almost at 100% coverage on their unit tests. High code coverage helps ensure bugs are captured on earlier, less “expensive” stages of SDLC.
Unit tests are most often written by developers, so this type of coverage is not often used by QA. But there is a way we can incorporate code coverage into the QA world.
QA Test Code Coverage
QA Test Code Coverage (a term I just made up) is a better way to measure Test Coverage for QA tests. It’s a code coverage of an application, running live in a deployed environment, while QA tests are being executed against it. Not Unit tests, but QA tests(regression, performance, etc.). The steps to get this report are a bit more complicated than with unit test coverage. You need to instrument your code. Then you need to deploy the instrumented app into an environment and after it’s deployed, you run your QA tests and this means whatever tests you want to measure coverage on regression tests, performance tests, manual or automated tests. This type of coverage has the same advantages as code coverage does(it’s objective, accurate, and automated). It can be used to see if your coverage is going up or down daily. And it can be used by QA teams.
The way it works is it’s logging the application call stack while tests are executed. A very simple example: given your application supports some login functionality, when you run a test that logs a user in, then the lines of source code that implement that functionality are called and the instrumentation tool logs those calls. If you run all of the regression tests then you should ideally expect that most of the application source code will be triggered and that fact will be reflected in the coverage report. If some lines have not been triggered, then you need to investigate why. Either you need to write some more tests, or there is some source code that is no longer used at application runtime, which is dead code, and you want to create a product backlog item for your developers to remove that code.
QA Test Code Coverage vs Code Coverage
The thing that I didn’t mention is that unit test coverage and QA code coverage analyze the same source code. So why bother measuring QA coverage, if both things cover the same code? Well, let’s quickly talk about why do we have these folks called QA engineers, and why do QA engineers do their testing. Here I will use an example from a presentation by James Whittaker (Testing Director at Google and later at Microsoft), who was talking about how Google tests software. If we think of software as a forest, in this world developers grow trees. In their vertical, developers water the trees, take care of them, and write unit tests for them.
Can there be bugs at the unit level? Sure! Can QA help writing unit tests? Possibly, but it’s generally the dev world. Real interesting bugs appear when trees are integrated into a forest though.
The dev may think like: hey, my tree is beautiful, it has great beautiful leaves…but look, it’s upside down!! So who is going to find those bugs? Definitely not devs. Look, they are still watering their trees, all fine. So this is where QA tests like integration tests, system tests, end-to-end tests come in. And even though, when we use QA Code Coverage for these tests, they will show coverage of the same source code, the key is that they will show that coverage in the context of different tests. So if some code was missed by these QA tests, it means that some functionality was not tested and that when tested, it can potentially reveal things like a good-looking tree, growing in zero gravity, and levitating or growing upside down.
To summarize the main benefits of QA Test Code Coverage. What are they? QA Test Code Coverage numbers are easy to calculate and to understand. It’s easy to automate the process of getting these numbers and analyze if coverage is going up or down.
If you use code coverage for your QA tests you will be able to find functionality that you haven’t tested yet and that can potentially have some bugs in it. Or you may find some dead code, which needs to be removed from the source code if it’s no longer used.
QA Test Code Coverage Problems
A little bit about difficulties with these metrics.
- Covered != Tested
Code Coverage is not enough by itself. The quality of tests will not be reflected in the coverage report. That’s the first problem. It will only show lines of code that were executed when tests ran. It’s up to the tester to make sure that appropriate verifications were done, that actual and expected behaviors were compared. - Functionality not coded for
Another potential danger is to fall into the misconception that as QA we only need to verify whatever developers coded and if our tests covered all the code, then all is fine. This part is called verification. Verifying actual behavior vs expected. Code coverage can help with this part. But as QA we should also do the validation part, validate the requirements. Are the product requirements valid? Should we change them and code something different, is there some important functionality missing, etc.? For example, in our forest example, did we also design worms that will break down leaves into the soil. And the best tool for validation is the QA mindset and thinking like a user, like a customer. - Can be difficult to implement
Another difficulty is that the implementation part can be hard. It’s a more difficult concept than unit test coverage. It varies from tech stack to tech stack significantly. And the fact that the coverage needs to happen while an application is running live in a deployed environment adds even more complexity. In our team, we used Istanbul to instrument a Node.js microservice running in a Docker container. The app was configured to be up for the duration of the automated test run, then shut down to report code coverage metrics to SonarQube. - Ignoring Code
Some lines of code will never realistically be hit during live app QA testing and, as a result, will show up as not covered in the final report. (Things like guard conditions, or error handlers for 3rd party services being down). Most coverage tools provide a way to mark lines or branches of code as “ignored”. Thus, you won’t need to come back to them after every test run to investigate why they were not covered. - Performance
Code coverage tools may affect application performance. It is never a good idea to enable the “instrumentation” mode in the Production environment.
