Quality: an often misunderstood concept
Many years ago, I wrote a tongue in cheek article that ‘mediocrity is underrated’. (it was written around 2011). It got posted on hacker news, and you can read that discussion here. It will read as being very cynical, and it sure is. But there is one part of it the article that deserves some thought
💡 You often want to be as good as absolutely necessary, but no better.
I expect most of us might not either agree or understand the intent of this statement. To explain what I mean, we need to understand what quality is. The pioneer in Quality was Dr. W. Edwards Deming. He pioneered approaches to quality control, and is revered particularly in Japan — his teachings resulted in the rise of high quality cars made by Japanese car manufacturers.
My two word takeaway from Dr. Deming on what quality is:
📖 Meets Specifications
That’s it. If you have met specs, you have delivered quality. You know why Toyota took such a lead in the 80s? They met specs better than anyone else. That’s it. Customers would refuse to take delivery of cars/parts not manufactured in Japan because ‘meeting specs’ with smaller variance meant that the cars or parts would just be more reliable, for a longer period of time.
Expunge words like ‘perfection’ from your vocabulary. Perfection has nothing to do with being perfect (Seth Godin). Perfect means meeting spec.
Of course, this means that the specs need to be defined. If there are no specs defined, then the question of ‘have we delivered quality’ is meaningless. There is no framework to use. Almost with certainty, not having specs means that you would not have met expectations because they have not been set. So if you find yourselves in a place where there are no quality specs written and agreed to, the question of quality does not arise. We deliver what we deliver. Which is unlikely to meet expectations, because there were no specs instantiating the expectations.
So what are the specifications for Software? They are generally covered in our non-functional requirements: see my writeup on Operational and Engineering excellence here, but in a nutshell
Operational Excellence is defined as the work done before and after code is deployed into production to maintain or improve operational quality.
Engineering Excellence is defined as the work required to create, deploy and maintain high quality products into production with speed and agility.
The goal of the specification is to help us improve continuously and demonstrate that we are getting closer to these outcomes.
Should you be beating Specs?
Once you have met specifications, going beyond that is of questionable value. Let’s say that our code coverage criteria is 70%. What is the incremental value of going to 80%? To 90%?
In some cases (e.g. the most often called module/function in the system or the place where we cannot afford to screw up) it is worth it. But for the rest: increasing this number will be a lot of additional work for relatively low value. Would we not have been better off moving to next project to unlock more customer value?
🔥 Takeway: Beat specs ~10% of the time. Meet specs 90% of the time
It is specs all the way down
The software we deliver is based on specification we get, typically from the product team. So how do we measure the quality of the specifications produced by the product team?
Engineering often has a binary notions of ‘correct’ and ‘incorrect’ tied to it. On the product side, it is way more squishy. We talk with customers & partner teams, understand behaviors and expectations and then write a spec (typically a Product Requirements Document, a PRD). How can we measure the quality of the PRD? It is not easy, and we need to agree on it, iterate on it, put it in front of customers and then test whether we built what the expected. This is why we need A/B testing and rapid iteration as we understand better. How do we know that what we built is ‘easy’ for the customers to use? How do we have a specification for ‘intuitive’?
In general, adjectives like ‘easy’ are inappropriate for putting in a spec. There is no way to measure ‘easy’ or ‘fast’ or ‘simple’ — there have to be specific metrics associated with it.
Company specifications
Most companies have OKR’s — an annual goal that documents the planned outcomes at the end of the year.
Of course, these specs can be somewhat unstable. The Business climate changes. Competition changes. Customer expectations change. The specs need to address these changes. The CEO at my last company used to quote something attributed to General Dwight D. Eisenhower
🔥 Plans are nothing. Planning is everything
stated otherwise, ‘No plans survive contact with ground reality’. But it is important to plan anyway and have a north star.
In Conclusion
Quality is about meeting specs. Choose to beat them infrequently. Choose to meet them always.
Engineering specs can be quite precise. It gets harder to be precise as you move away from a rigorous practice of engineering to more of the art of product and predicting company outcomes
Practice continuous improvement — and measure the improvement. The goal will take care of itself if we are practicing and demonstrating continuous improvement.
