Software Quality

Welcome back! This year you can expect more research, interviews, and benchmarks on developer productivity. We’ll also host more events, like this one next week on DORA, SPACE, and DevEx.

This week I’m sharing the latest paper in the “Developer Productivity for Humans” series by the Engineering Productivity Research team at Google. In the past they’ve covered topics such as managing technical debt and the impact of improving builds times. This paper defines code quality and explores the factors impacting it.

My summary of the paper

Google‘s Engineering Productivity Research team subscribes to the belief that no single metric captures developer productivity. Instead, they break developer productivity down into three dimensions: speed, ease, and quality. Anytime they’re measuring one of the three dimensions (for example, how long it takes for code reviews to be completed), they also measure the other two to surface potential tradeoffs. 

However, while speed and ease are more straightforward, quality is difficult to measure because it is difficult to define. Quality means different things to different people. At Google, this became an important question to answer. 

To better understand what “quality” means, the research team conducted two series of interviews with developers at Google to explore code quality and product quality. Developers were first asked to define each type of quality. Then, they were asked to describe how quality impacts their own productivity, and which factors influence their satisfaction with code quality and product quality.

The four types of software quality

Based on the interviews and previous research, the authors developed a “theory of quality.” This theory posits that there are four types of quality that influence each other. These are defined below, along with commentary on how they impact each other and how they may be measured. 

1. Process quality 

• How it’s defined: A high-quality process includes things like having comprehensive and deterministic testing, thorough code reviews, organizational consistency, and an effective planning process. 

• How it influences other types of software quality: There’s good evidence from prior research that process quality can predict overall software quality: for example, multiple studies have shown that process-based metrics are more predictive of post-release defects than are existing code quality metrics. That’s why in this theory, everything begins from a high-quality development process. 

2. Code quality

• How it’s defined: The researchers found through their study that developers define code quality as primarily relating to maintainability. Maintainability has subcategories including testability, comprehensibility, complexity, and readability.  Overall, developers described code quality as “the ease of working with and understanding the code so that they can easily make changes to it.” 

• How it influences other types of software quality: The researchers found that code quality improves the system quality by reducing defects and increasing reliability. High code quality also leads to higher velocity for developers. 

The next two categories are typically what executives and product managers think of when they hear “software quality.” While developers think about their code and process quality, executives think about system and product quality. (The researchers note that this can cause a disconnect.)

3. System quality

• How it’s defined: System quality is characterized by its reliability, performance, and defect rates. 

• How it influences other types of software quality: Having high code quality is a necessary requirement for having high system quality.

The authors add some interesting commentary on system quality: specifically, that it’s difficult to measure because of the sparsity of data. Outages are rare, so even when there are no outages for a period of time, it’s difficult to tell whether system quality has truly improved. For that reason, process and code quality metrics are useful for tracking the indicators that determine system quality. 

The authors continue, explaining why these metrics are useful for understanding system quality: “Without these intermediary metrics, stakeholders may think that a year with no outages is evidence that the system is high quality… In a year with no outages or incidents, two realities could be at play. In the worst-case scenario, developers may have been effective at inefficiently working around various weaknesses to prevent bugs in a poorly operating system. In the best-case scenario, developers were working in a system with a low risk of defects and were free to focus on enhancements and iterating on new features. Without code quality metrics, leadership can’t be sure, and they won’t know how to direct engineering efforts to ensure developer velocity and product quality over the long term.” 

4. Product quality

• How it’s defined: Product quality is defined as having three key components: utility, usability, and reliability. 

• How it‘s connected with other types of software quality: Product quality and code quality are connected: low code quality can slow engineering velocity and consequently delay product improvements or even render them infeasible. Lower code quality can also increase the risk of defects, which also influences the quality of the product. 

In their conclusion, the researchers suggest that if a team is focused on improving software quality, they should determine which type of quality they want to improve first. This will determine the metrics they use and the actions they take. 

Final thoughts

In a previous issue, I shared a study from Google which found that an increase in developers’ satisfaction with code quality leads to an increase in their productivity. This new study helps us better understand both how we can improve code quality, as well as how code quality relates to other types of quality (which product managers and executives tend to care more about).

It’s interesting to think about how this definition of software quality can guide the actions that teams take to improve. If software quality, product quality, and code quality are all influenced by having a high-quality development process, then this gives leaders a good starting point to begin taking action. 

Thanks for reading. Subscribe here to get future issues.

-Abi