100% of Test Cases are Covered by Automation, Is That Important?
As Software QA professionals, we often find ourselves fixated on the idea of achieving 100% test coverage through automation. In some organizations, this has even become a measure of success in the QA profession. Let’s explore this concept together.
So, what does “automation test coverage” mean? In simple terms, if a testing team executes 10 test cases for a particular feature and then automates these 10 test cases, making them run automatically at both the API and UI levels, the automation test coverage is considered 100%.
The next question is, what are we trying to convey with this 100% test coverage by automation? What message are we aiming to deliver to the organization?
Having 100% automation test coverage doesn’t always mean that an application or feature has been thoroughly tested or is ready for production. What truly matters is the quality of the test cases themselves. Are the automated test cases truly covering a variety of testing scenarios that could occur? Are they relevant and representative of everyday user behavior? If automation is pursued without considering the quality of the test cases, then 100% automation test coverage can be an empty number, especially when features or applications evolve, requiring periodic updates to the automation suite. Too much automation that is difficult to maintain can become problematic, making the already 100% automation test coverage meaningless.
What I am trying to convey is, that we can focus on numbers such as automation test coverage, but we need to see first, how mature our testing team is, if one QA person makes test cases alone, and there is no one to review the test cases. and executing the test case itself, then the test case becomes very subjective, when the test case is automated, the automation test coverage number becomes a subjective number, and yeah, the number does not mean anything.
Why do I say that test cases become highly subjective? Different individuals may have different understandings of what should be tested in a feature or component. Testing criteria set by one person may not cover all relevant aspects, and different individuals may take different approaches to testing, focusing on different testing scenarios, resulting in different test cases. So, instead of fixating on automation test coverage, the process of reviewing and validating test cases from various perspectives becomes more critical.
However, the challenge in this agile software technology industry is that testing is often overlooked. There may not be much time for reviewing and validating test cases before execution, the test case is entirely QA’s responsibility, and QA must be able to provide test coverage answers, that are actually meaningless.
For me, rather than obsessing over the elusive 100% test coverage, it is important for a QA professional to take a balanced approach. Prioritizing critical features and essential areas of the application for in-depth testing while automating repetitive and time-consuming tests can generate significant benefits. Instead of getting busy with these complicated test coverage metrics, if such metrics are necessary, we can measure the success or professionalism of a QA professional by the number of bugs found in production. The fewer the bugs found in production, the better, indicating that as QA professionals, we have fought hard from the beginning by clarifying requirements during planning, planning testing rigorously, executing it, and automating it.
“Utopia 100%” in automation testing is an admirable goal but should be approached with a pragmatic mindset. A balanced approach that prioritizes quality, efficiency, and effective results in software testing is the key to achieving meaningful goals in QA. This, I believe, is what a QA professional can contribute to an organization. Of course, these contributions can also be measured by metrics, but certainly not by automation test coverage.
This is my perspective. What do you think? 🍻
