Quality. Without Testing.

Quality is not just about testing despite the stereotype. Besides, is it really up to QA to do the tests?

We set about working on this theme during our last round table “Making Quality. Without tests. ”. This article summarizes the main points we shared.

Software quality is still too often found only associated with testing. Real challenges of value perception, communication, and collaboration are at the heart of this issue.

I thank the participants for their presence and contribution:

Arnaud Dutrillaux, Senior Quality Engineer at La Javaness
Benjamin Butel, Agile Coach at Klaxoon
Christophe Moustier, Quality and Test Expert at Inetum, Author
Farah Chabchoub, Head of QA at Livestorm & Speaker
Iman Benlekehal, Specialist in Software Quality Assurance
Olivier Dennemont, Head of QA at Manomano

Join the QE Unit to access more exclusive content from the community.

Quality begins (grows and ends) with culture

“Tests? You have to see with the QA team”, a phrase that we regularly hear, without necessarily counter-arguments. This myth is widely held and accepted in the software industry. Codes, beliefs, and shared practices are the basis of culture. The actors act in accordance with their culture to define, build and maintain software.

Achieving quality, therefore, requires an evolution of the culture of the organization.

Like some tribes that have developed entirely different codes, some organizations value quality differently. Spotify is a good example that has resulted in a successful, scalable, and sustainable product approach. Benjamin drew a parallel between the level of quality and our choice of tangible products: low, mid, or high-end.

“You have to discern what quality of the product you build, up to the management; do you want a disposable low-cost, a cheap for two years, or better?”
Benjamin Butel

For other cultures, from startups to the Fortune 100, quality is often a low-cost model; we develop every two years again, keeping and accumulating technical debt until the next wall. These cultures often associate QA with end-of-line tests once the developments have been carried out, without business contact and even less with the customer.

This segmentation of QA is one of the main issues for a more transversal quality. Olivier shared his personal experience with us where the direct contact with the product users – particularly their feedback – made him aware of the quality imperative.

“The proximity to the users made me aware of the quality of the product very early on.”
Olivier Dennemont

This proximity reminds us that software is not just code; it is a solution that must be useful for someone. This empathy is necessary; we can easily lose sight of the customer while immersed in our technological questions.

Coming back to culture, the established codes and words are powerful and revealing. Is the QA team materialized by an organizational silo in the organizational chart? Does the team leader have a “Lead Test” title? Do the team members talk about tests, quality, or business value?

These are all variables that influence the perception of the quality of the organization. Quality is therefore structuring, starting first with ourselves; Are we talking about the business? Are we in contact with the customer? Do we work to decline the strategic priorities on the axes of quality?

The work begins by ourselves before to broaden our impact on the rest of the organization. Cultural changes take time, and for this reason, they must be initiated early.

The first step is to build a shared vision through collaboration.

Co-constructing a common corporate objective, supported by quality

The collaboration effectiveness is supported by common company culture and shared goals. Without this star, local forces will gain the upper hand to the detriment of the system’s overall performance.

Ensuring quality must therefore be a common unifying objective; a track, it is not software quality as such. Let’s take a step back before talking about Shift-left, Shift-right, DevOps, microservices, or Holistic Testing.

Applying the “First things First” and “Start with Why” principles are relevant here. Iman shared the term “Shift-up and Spread” to start our thinking, supporting a focus on the company and its customers before widening our prism.

“Before talking about Shift-left or Shift-right, Quality must identify the value in Shift-up.”
Iman Benlekehal

We come back to the initial “Why” of the organization through its mission, vision and values. This motivation of existence must translate into internal and external objectives, particularly for the customers. The breakdown of quality imperatives must derive from this raison d’être, declined at several levels.

First, our approach must involve the various stakeholders in collaborative workshops, from sponsors to operational staff. Iman shared with us the key objective of aligning quality priorities beyond initial individual perceptions. Farah has also shared its practices in this interview: Quality Assurance, From Operations to the Board of Directors.

Second, the quality value proposition must bring the business vision as close as possible to that of IT. The rise of the digitalization of companies is an opportunity to be seized in the articulation of quality value. We can find elements of excellence in the customer experience, business development iterations, or the reliability of our corporate backbone.

Models have emerged from major players who have had to address similar issues, moreover, at scale. Google, for example, has changed its concept of Site Reliability Engineering (SRE) towards Customer Reliability Engineering (CRE). This interesting turn shows the transversalization of practices and the strength of the words chosen.

Our value is to articulate a quality value proposition by identifying appropriate choices in our context. We have a strong advisory duty to bring out the invisible issues of the business and the trade-offs they involve. Arnaud shared the need for integrated quality, especially in dynamic environments.

Accelerating delivery cycles requires transversal and inclusive quality. There are no other options. ”
Arnaud Dutrillaux

Lately, the quality that we want to co-build requires reinforcement mechanisms, known as feedback loops. This is where the inclusion of sponsors and influencers is more than relevant for communication, arbitration, and quality defense actions, especially when the pressure increases. Christophe mentioned the importance of double learning loops that we will detail afterward.

Sponsors are also needed to change the organization, a structural element of quality improvement.

Moving from quality assurance to quality assistance

An organization chart alone will have little impact on the dynamics of an organization and its activities. The famous idealized reorganizations revealing themselves to consume time and without any improvement are an example of this. Individuals, their interactions, and their objectives are at the heart of the organizational system.

So let’s come back to the positioning of quality. We often find a siloed QA team called upon somehow in different projects and ensure this famous quality. This mode of organization will tend to consult the QA once a need for BDD, test, or other is felt. It is far too late to make quality, let alone deliver the value of quality upstream.

“Carrying out test activities reinforces the feeling of quality ownership.”
Olivier Dennemont

So we just have to make a Spotify-like model, including horizontal QA profiles, on nice slides of agility at scale for our sponsors. These models must be adapted to the system as a whole: the company’s culture, the organizational and product architecture. In addition, the horizontalization of the function reinforces a need for vertical animation of a practice; if we want to keep a minimum of consistency.

Unfortunately, even if these models influence the dynamics of the organization, they do not go to the heart of the problem: doing quality requires structural inclusion in the different processes. We can also speak of the concept of capabilities for this purpose.

Doing quality implies empowering actors other than QA on quality, not just on slides. We must act in reality by changing the interactions, the positions, and the types of activities carried out.

“At Manomano, the QA teams no longer do testing for the other teams; we are a quality assistance team.”
Olivier Dennemont

Integrating quality seems obvious. Still, how many of these scale agility reorganizations include a culture change plan, shared vision, and role reallocation for integrated quality? Not the majority, given an ecosystem that continues to associate quality with tests. Adaptive models are emerging, such as X-teams.

A concrete consequence is the evolution of the role of product and engineering managers having to consider a more global quality; it is a good subject for organizational development. Some models advocate the disappearance of managers. It is not necessarily the subject if they are replaced by coaches in craftsmanship or the latest fashionable collaborative feedback tool.

This concept of Quality Assistance materializes more clearly the responsibilities expected of each team. This is not the only concept pushing for more accountability of cross-functional teams while providing them with tools: we can mention engineering productivity or the developer platform.

Moreover, the acronym QA has another variant available in the references, “Question Asker”.

Allowing the prioritization of quality at the heart of the processes

Since this section begins with the word “prioritize”, we can ask ourselves the link with our “Question Asker”. Making quality as a QA will increasingly require empowering product teams and frontline players. Relevant and open-ended questions will support the teams in their inclusion of quality.

We often wrongly estimate our understanding of a subject; questioning in all humility is already a quality practice in itself. If we also want to impact without being in the limelight, asking questions with added value will demonstrate the interest in interacting with quality teams.

Integrated quality must also be achievable. Accelerating deliveries, product complexity, and organizations require a scalable model of quality inclusion. Adding QA profiles doesn’t scale, in addition to being inefficient. Therefore, priority must be carried out directly by the players working on the product; quality teams are there to assist their approach, prioritization, and implementation.

“We’re here to help teams test upstream, on the way they work, not on the finished product.”
Olivier Dennemont

Companies’ resources are limited in the face of the challenges and objectives they wish to achieve. That is the reality. Prioritization is essential and found in the concepts of Work In Progress and LEAN. The value of initially going through a “Shift-up & Spread” takes on all its importance here.

For some teams, a Shift-left approach focusing on BDD may be the right choice in their context. Conversely, CRE and support improvements will be for another team. The context is fundamental in the continuous exercise of prioritization; the latest models and fashionable acronyms are to be seen in a second step or to keep for the coffee break.

Collaborative prioritization will be materialized by actions to be taken, often in the context of sprints. The subject of task assignment can start without remaining the private hunt of QA.

Composing in iterations by measuring progress

The previous steps consisted in securing the “right product”, leaving the equivalent challenge of delivering the “product right”. We will test our model and our assumptions in search of value creation with quality through the iterations.

Our iterations must stay focused on the initial common objective of creating value for the company. Maintaining direction is a real challenge during execution cycles where you can quickly lose perspective. Measurement is essential to verify the adequacy of our actions, but which metrics to use?

Indicators are relevant when they are taken in their contexts, such as the organization and its priorities. For quality, we have retained the use of double learning loops: that of the creation of value (“right product”), and that of the improvement of the system producing this value (“product right”).

The choice, interpretation, and communication of indicators present real implementation complexities. We can suffer from our cognitive biases, lack business relevance, and have difficulty demonstrating a correlation of our actions. The subjects of Observability, Value Stream, and Analytics are to invest throughout the process.

Pragmatically, our recommendation is to focus primarily on measuring the business value commonly sought.

Another recommendation is to spend more time actually trying to create value than trying to measure (test?). Iman pictured it very well on the example of blood tests: doing more will not guarantee good health.

“Doing more blood tests (testing) will not guarantee good health (quality).”
Iman Benlekehal

Reality is not all about success, and while we would always like to share examples of winning cases and early wins, sometimes we must recognize we hit a wall. We find this notion in Fail-fast, interestingly by the pressure put into validating the structuring assumptions before going further, combining measurement of the value and reducing the risk.

Walls make companies react, but our goal is to avoid them as much as possible and reduce their size. Communication is therefore key to convince as early as possible.

Communicate, Communicate and Communicate

The term DevOps is so widespread that we even forget the colossal communication effort that has been put into it. Our quality will become an integral part of our organization, culture, and processes with systematic and holistic communication. We usually have to plan ten times more than we imagined.

However, quantity does not rhyme with quality, and counter-intuitively, communicating about quality is not just about quality.

Communication in the service of transversalization of quality must be omnipresent, without necessarily being directly associated with it. The indicators of value creation for the company are a good example; their mere presence is already a success of more quality. Their usage, constant reminder, and inclusion in feedback loop mechanisms will support their valuation.

The important thing is to secure the frequency, coverage, and inclusion of the various stakeholders of value and quality in our actions. We need to know how to talk to a management and the hour after, with a development team. Transversal communication will support a transversal quality.

Our priority is to bring out quality criteria during development cycles through our communities, interactions, and Question Asker’s role. Arnaud shared his experience where the identification of requirements raised awareness. The presence of various quality topics is the first step for them to be considered. The rest of the work lies in the culture and the ability to support the achievement of results.

The approaches with the most significant impact on the organization are, by nature, transversal. Take the example of our SRE transformed into CRE combining user experience, engineering, and operations. The addition of error budgets will balance the need for acceleration and stability of the experience, another element of an inclusive quality.

In reality, built-in quality becomes natural and transparent. It can be a frustrating and disappointing side if we were looking for fame instead of quality.

Quality needs testing, far from being limited to it

Making quality without testing is therefore possible, bearing in mind that testing remains necessary. Testing is not the exclusive subject of quality nor at the heart of the matter. We actually have real challenges for the evolution of cultures and organizations in the ecosystem.

“Making Quality is first and foremost driving organizational change.”
Antoine Craske

Olivier and Arnaud shared their models of a QA as a real lever of quality for the rest of the organization, not directly dealing with the tests. More transversal organizations are therefore more suitable if we collaboratively address culture, alignment, and other shared mechanisms.

Cultural changes at scale take time; entire generations are often necessary to verify structural changes. Let us hope that our efforts will enable us to accelerate this transformation of the quality and capacity for innovation of our companies, for those which will survive.

Again, the perspective we take makes all the difference. At our level, we must transform our organization for integrated quality. On a more global scale, the whole digital ecosystem must evolve until it reaches its Tipping Point to transform the culture of software quality more broadly.

So let’s work on this shared mission to have the chance to experience this transition.