The decision is taken, “we are going to address our test debt”. It sounds positive, but at the same time we wonder, “why did we end up at that point?”.
Test Debt is a waste that organizations must continuously prevent and eliminate to maintain their speed and reactivity. Else, each change comes with its added weight, slowing down the overall software delivery.
During the round-table “Forget Test Debt: Build Test Automation Wealth” we shared the definition and key reasons for test debt before sharing actionable ways to build test automation wealth in the first place. This first article focuses on Test Debt, while the next one on creating test automation wealth.
I thanks each of the participants for their participation and contribution:
- Joel Oliveira, Software Testing & Quality Assurance Evangelist, Public Speaker, Trainer, Senior Program Manager, Independent Consultant
- Rafael Amaral, Senior Software Engineer in Test at Farfetch
- Filipa Nogueira, Engineering Team Lead at La Redoute
First things first, let’s start by exploring Test Debt with a concrete example.
What is Test Debt anyway?
A company judges too slow its software changes iteration. Test automation is identified as a priority to accelerate the cycles. A framework is quickly built to automate the existing manual test and produce early wins. Then problems start to appear.
The precipitation led to an automated test suite not working well: many tests are executed with slowness and instability, while it is hard to implement some manual tests. In addition, the only person knowing the framework will leave the company. You open the tests to try to help but they are too technical and hard to understand.
You hoped for much better results. A fast and reliable automated test suite on the most valuable tests for the team. A framework known, maintainable by other persons or a vendor. A built-in reporting on shared team objectives and measures.
Technical Debt is about: Expected Asset Value – Actual System Value
Technical Debt and Test Debt share the common characteristics of being a gap between an expected and actual value. The Test Debt can be due to the lack of specific actions or be the result of particular decisions. It is important to keep these two typologies in mind as counter-intuitively, both inaction and action lead to debt.
The presence of Test Debt leads to the next question, why did we get there?
Why do we end up with Test Debt?
We can start by looking out of the world of software and testing, especially in finance, reusing our analogy. Why do so many people endup in a financial debt situation? Common reasons are shared with Test Debt.
A first point circles back to the two elements of measuring test debt, “expected” and “actual”. The expected value means that it was defined, shared, and agreed upon. Unfortunately, the quality expectations are not yet a reflex of collaborative exercises, such as the Shift-Up one. The “actual” value lies in a characteristic of Test Debt: it is hard to measure.
“The tricky thing about technical debt, of course, is that unlike money it’s impossible to measure effectively.”
Martin Fowler, at martinfowler.com
Financial debt is easily measured; you owe a specific amount of money to an entity. For technical and test debt, that is less easy. You can use the quality attributes of tests, technical debt models, and link them to essential indicators. This is a first step. Their technicality and lack of alignment with the business drivers can make them hard to understand for other people.
Another key factor lies in a cocktail of 3 dangerous factors :
- Focus on short-term
- Lack of understanding
- Delayed feedback loops
Short-sighted approaches come at the cost of piling up debt and side effects in the mid-term. Stakeholders can push in that direction to meet specific objectives, show their power, or just also not being conscious of the trade-offs, the second factor. Our work in quality is to express the associated consequences of specific choices. Not doing that results in the third factor, delayed feedback loops, that are hard for humans to anticipate.
We can make an analogy with a common situation in human behavior. A child starts to eat some candy. He feels good, then he eats more of them. No one told him about the impacts of candy over time in the body, mind, etc. The children do not care as being young its body processes quickly, in the short-term there is no problem. Over time, he starts to gain various pounds until he reaches an obese critical state and has to go for surgery.
This is how these 3 factors combined lead organizations to hit a wall, not able to deliver software changes anymore. Then, they have to implement drastic changes if they are still able to survive on the market. Test Debt does not happen overnight. It is the sum of taken and non-taken decisions by a set of actors within the ecosystem.
We identify an essential practice relying on intuition and data, the warning signs.
What are the warning signs of Test Debt?
Test Debt is hard to measure factually, but we can rely on our human capacity to detect, feel and react to warning signs. For test automation, we can sense organizational behaviors and specific test automation attributes.
Let’s get back to the Why of our automated tests. One objective of our test automation effort is to accelerate the delivery of software changes with confidence. The test automation value disappears when the team starts to bypass the test automation campaign, search for alternative routes, ask for exceptions. Various reasons are possible as a long execution time, instability, lack of understanding, or other maintainability criteria.
The execution time is directly tied to essential indicators of software delivery: lead-time for changes, cycle-time, and MTTA. These metrics are all part of the Accelerate report, correlating the organization’s performance with these measures. We need to constraint our test execution time to limit its impact on these acceleration metrics. For test automation, it means less but more valuable tests executed faster. It can also be about decoupling the tests to be executed under a certain perimeter instead of running each time a large end-to-end regression campaign.
The second factor of test instability, also known as flakiness, is measured with the flaky ratio. Flakiness is not impacting only the test. It directly impacts the trust of delivering software for your team. It is therefore essential to aim for that percentage of 100%. When this is not the case, understanding and fixing the root causes is a priority. Flakiness can be measured when you have at least one automated test implemented, checking its stability over time. When adding tests, you can also start to measure the overall test suite stability.
The last factor focuses on test automation maintenance warning signs. The key is to act early on specific indicators and behaviors. Regularly assess the impact of the automated test by asking “So what?”. It aims to confirm that the automated tests are having effects on proprietary topics. Are you able to show this type of correlation? If you hear sentences like “I don’t understand what QA is doing anyway”, this is a warning sign. The team can be lost in technical optimizations.
The danger of lacking an alignment is the one of local optimization to the detriment of the whole system. It is dangerous to let a QA team focus on extensive rework, framework optimization, test redesign. They start to take the habit of optimizing their feedback loops without necessarily providing value to the rest of the team. This type of behavior leads to satisfied technical persons completely forgetting about the true goal of their work. Taken too late, you can hear “It’s better to start a new framework, it became too complex.”. This leads us to poor test design.
Automated tests require design like any proper engineering activity. A number of pitfalls must be avoided, like copying the tests from a manual suite or letting a recorder create tests. You can decide to use some of these assets once the preliminary works of architecture, design and modularization have been completed. Test design is also about selecting which tests to don’t implement, an important typology of test to clarify from the start.
The trend of these symptoms is one of unbalance. It is when the costs/benefits equation is not satisfactory anymore.
Smash Test Debt Before It Is Too Late
Stakeholders are people that invest money for two reasons, to win or stop losing money. That investment can stop when the test debt comes to an unacceptable gap of actual versus and expected value.
“Test Debt does not happen overnight. It is the sum of taken and non-taken decisions by a set of actors within the ecosystem.”
Antoine Craske
We all want to avoid this situation. Covering the reasons leading to test debt clarifies that the multiple factors are not due to a single person, team, or problem. A more integrated and incremental approach is required.
We cover this approach in the following article about empowering test automation with quality engineering.