How to tell if your DevOps is delivering business value

DevOps, where is your ROI?

That's the question pondered in a recent interview posted at DevOps.com, in which Carol Carpenter, CEO of ElasticBox, remarked that the retrun on investment for DevOps is still an inexact science. "What are we investing? How are we spending our money? And what is our return? The ROI analysis I think is still very early and immature around DevOps," she commented.

In the discussion with DevOps.com editor-in-chief Alan Shimel, she observed that DevOps teams tend to be spread very thin, and "are being asked to do work in what used to be traditional IT ops, in addition to work that's now kind of develop or deployment work, and also be a communication bridge across these different groups."

Both business and IT leaders understand instinctively that DevOps is a good thing, but the measurements are still elusive, Shimer agreed. They're signing off on DevOps "if their gut tells them that automation is good, that DevOps can help, that more releases, higher-functioning IT directly results in higher profits, and higher productivity," he points out. "But the bean counters haven't jumped in there yet and done the real statistical analysis, to really show us the ROI, the return where you're going to get it, and this is how much more profitable."

DevOps is seen as the best approach to align the creativity and innovation of developers with the increasingly fast-paced delivery and implementation cycles required by businesses. With so many resources going into this kind of an effort, and with so much at stake, measurement is critical.

Peter Waterhouse, CA's resident DevOps guru, in a set of guidelines, has attempted to identify some of the key metrics that determine the success of DevOps in delivering business value. The important takeaway from Waterhouse's analysis is that successful DevOps means more than simply getting software out the door faster, or sticking to some kind of schedule. Successful DevOps should be measured on the basis of what value is being delivered to the business. Is the business able to compete more effectively? Is it delivering more value to customers? As Waterhouse puts it, the success of DevOps isn't just about measuring "velocity, defect counts and function counts without paying attention to delivery, customer satisfaction and loyalty."

Here are the metrics that matter when it comes to measuring DevOps success, as explained by Waterhouse:

Customer and business value: These metrics focus on what the business is gaining, "helping measure the impact of DevOps on meeting business goals, like increased customer loyalty and faster time-to-market," says Waterhouse. "The manufacturing concept of lead time provides DevOps practitioners with an analogous metric (time taken from when code starts development to successful production deployment) that determines how well DevOps is meeting the need for rapid delivery of high-quality software services. This metric is especially important because long lead times are often indicative of high levels of code defects, testing constraints and contentions."

Culture, collaboration and sharing: "Metrics in this category are the cultural bedrock upon which a DevOps program is based," Waterhouse states. "They are especially valuable because they provide an ongoing indicator of acceptance/resistance to new DevOps thinking." Metrics in this category may include staff retention rates, turnover, employee morale, responsiveness to change.

Efficiency and effectiveness: These are the metrics that focus mainly on conventional operational issues, such as server to admin ratios, transactional costs, data center efficiencies, and cost of release. It's also important to adopt more customer-centric ratios, as well, Waterhouse urges -- such as full-time employee counts to customers.

Quality and velocity: Metrics in this area focus on service delivery such as percentage of deployments rolled-back due to code defects/outages/negative user reactions. "Metrics like this can also provide additional insights when paired with other indicators," Waterhouse points out. "For example, if the rate of rollbacks increases during periods of low-change volume it could be indicative of more serious problems -- errors due to manual/scripted release processes, task switching and excessive handoffs." Other key metrics include cycle time and mean time to repair (MTTR), which he calls "a great indicator of how effective teams are in handling changes."