Your organization has invested time and money into automating a portion of its testing. You’ve built your framework, scripted your tests, and now they’re running and providing you results. Great work! You could simply leave them running and hope they’re doing what you intended. However, you’ve already put so much time into getting where you are, and will indeed have to spend time maintaining them well into the future. Instead of hoping they’re successfully giving you the right results, let’s talk about some metrics you could put into place to monitor and ensure they’re delivering value!
Why Measure?
Before deciding on what to measure, it is important to consider why you want to measure automation quality and effectiveness to begin with. Understanding the why upfront will help guide you in selecting the best key performance indicators (KPIs) for your organization. There are many reasons your organization could be driven to use KPIs:
- Automation Effectiveness: You’ve invested quite a bit of resources into getting an automation suite built. While the goal for automating your regression suite may have been unique to your organization, a fairly common objective is to deliver code to the customers quicker without putting quality at risk. KPIs can be used to track and trend effectiveness over time and help you meet that goal.
- Speed to Market: You wanted to speed your release timelines up and measuring that speed to market will help ensure you’re trending towards that goal.
- You want to create a culture of accountability: To hold someone accountable, you must first communicate their success measurements. Ensuring you have a standard set of KPIs, communicated consistently, will allow you to hold people accountable, and to do so fairly.
- Continuous Improvement: Getting better over time should be a goal for any organization. Measuring trends will help you adjust and overcome challenges as you improve your automation scripting and processes.
Pitfalls of KPIs
Before we jump into specific KPIs, let’s walk through some common pitfalls to avoid. Using KPIs for the sole purpose of informing and improving is ideal. However, using them out of context or applying them incorrectly, paints an inaccurate picture. Avoid these pitfalls when you’re implementing your KPIs:
- Don’t focus solely on one area: Even if your goal is to measure quality, your focus should be throughout the entire software development lifecycle (SDLC). Focusing in on testers, or only the testing phase, will not give you the big picture view of quality.
- KPIs aren’t helpful if taken out of context: As you’re considering what KPIs will be useful to your organization, and your ability to paint a whole picture, make certain you are deriving useful contextual information.
- Using metrics to spy on teams: Honestly, this is a common pitfall where we’ve all found ourselves. There are two ways to measure success: let teams use KPIs internally and self-manage OR have a set of KPIs that you routinely monitor to help guide customer satisfaction, process or tool effectiveness, or coaching opportunities. Do not fall into the trap of using KPIs solely to manage productivity. KPIs should be used as a starting point and lead to further conversations.
What Could You Measure?
Automatable Test Cases
What Does It Tell Me?
This metric can help you understand where your teams are prioritizing automation or what areas might still require manual validations. Breaking this metric up to cover specific components of your application can provide even greater value.
How Do I Calculate It?
% Automatable = (# of automatable tests / # of total tests) * 100
Automation Script Effectiveness
What Does It Tell Me?
This metric provides clarity on how your defects are being found. If you are investing large dollars in automation, but your automation scripts aren’t finding defects, you may want to dig in and understand the effectiveness of those scripts. Also, if you use different testing environments, like integration and staging, you may expect to see lower effectiveness in your integration environment because scripting hasn’t been completed for new features. Likewise, in your staging environment, you would expect to see automation finding the majority of the defects as you’re typically focused on regression testing.
How Do I Calculate It?
Automation Script Effectiveness = (# of defects found by automation / # of defects opened) * 100
Automation Pass Rate
What Does It Tell Me?
This metric will give you a standard percentage of how many automation tests passed. It is helpful in understanding the stability of your automation suite as well as its effectiveness. Having a low pass rate requires more time spent validating failures. If your failures turn out to be false failures, that’s an early indicator that your automation suite isn’t reliable. In addition, if you see this number dip after an automation execution, this might be a quick indicator that your release contains a higher than normal number of defects.
How Do I Calculate It?
Pass Rate % = (# of cases that passed / # of test cases executed) * 100
Automation Execution Time
What Does It Tell Me?
The metric will provide a value of how long it takes your automation suite to run from beginning to end. A long-running automation suite isn’t valuable when it comes to providing feedback to developers or delivering code to production quickly.
How Do I Calculate It?
Execution Time = End Time – Start Time
Automation Test Coverage
What Does It Tell Me?
Measuring test coverage will help a team understand how much coverage their automation suite is providing vs how much manual testing is being done. You can track this on a component basis and then summarize the data to visualize it across an entire regression suite.
How Do I Calculate It?
Automation Test Coverage % = (# of automation tests / # of total tests) * 100
Automation Stability
What Does It Tell Me?
This metric will help you trend how well your automation performs over time. It’s not uncommon for your scripts to fail, especially flaky UI scripts, if something in the system changes and automation isn’t updated prior to execution. However, if over time, your tests are continuously failing, that is a good indicator that your tests may not be stable.
How Do I Calculate It?
Automation Stability % = per test (# of failures / # of executions) * 100
Build Stability
What Does It Tell Me?
If you’ve taken the step of getting your automation into a CICD pipeline, you can use this metric to understand the level of code quality coming from development. If your builds are constantly breaking, there may be opportunity for better unit testing, for example.
How Do I Calculate It?
Build Stability % = (# of build failures / # of builds) * 100
In-Sprint Automation
What Does It Tell Me?
The metric will help you understand how close you are to in-sprint automation. To be most valuable, your team should strive to automate new work in the iteration the work is completed in, and not in a subsequent sprint.
How Do I Calculate It?
In-Sprint Automation % = (# of scripts created in-sprint / # of scripts created post-sprint) * 100
Automation Progress
What Does It Tell Me?
Regardless if you’re measuring this against the goal of automating a regression suite, or a new feature, this metric will assist you in trending how you’re progressing towards that goal over time.
How Do I Calculate It?
Automation Progress % = (# of automated tests / # of tests that are automatable) * 100
Automation Pyramid
What Does It Tell Me?
While there are several theories around how the automation pyramid should look, the outcome is a general understanding that a certain amount of tests should live at the UI/E2E, Contract and Unit test levels. Having a metric to continuously monitor that your testing coverage is following your pyramid strategy is important.
How Do I Calculate It?
Automation Pyramid = (# of tests at each level / # of total tests) * 100
Conclusion
There are a lot of options for automation metrics and as you read through each one, keep in mind why you’re wanting to implement testing KPIs in the first place. Once you identify what KPIs help meet your goals, make sure you’re communicating appropriately so people understand what KPIs may be used as a success measurement and why. Last, but certainly not least, use these consistently so you develop a trend to help identify anomalies over time. When implemented correctly, automation KPIs can help drive automation effectiveness, speed of delivery, and assist in delivering quality features to your customers.
The QA Connection Blog 
One comment