Continuous Quality — Shifting in ALL Directions (Part 2)
Several months back, in a previous article, I shared my thoughts and ideas about how to achieve continuous quality. At that time I focused on the much more commonly known ‘shifting left’ approach, and what it really means beyond having automation in a CI/CD pipeline.
Well, I’d now like to talk about the less commonly known ‘shifting right’ approach and some of the things you can try and implement too.
So lets get going….
What does it mean to Shift Right?
Shifting right is generally not something teams really think about when it comes to testing and quality. It is all about shifting left and how we can get quality products to production as quickly as possible. However this is where we can gain a huge amount of insights and benefit from focusing on what is actually going on in our production environments, and whether we have actually delivered high quality value to the customer. For continuous quality to really be achieved and measured we need to focus here also and take the learnings from what has been delivered….and there are many ways we can do this.
Testing In Production
Lets start with possibly the most controversial of the shifting right techniques….Testing in Production. Now, when I say testing in production, I don’t mean production is the first and only place you test changes/new features — well it could be in some circumstances, but that ideally shouldn't be the normal practice.
What I really mean is carrying out further testing or ongoing testing in Production to supplement the testing already carried out in lower environments. By doing so we can gain further insights that help understand if what has been delivered is actually the right thing and customers are using it how expected. But also, help us discover what would/could be the right thing to build in the first place. By taking those insights and learnings, we are able to feed them back into future developments & delivery cycles, and keep improvement and delivering customer value.
We certainly shouldn’t be afraid to test in production. We should in fact encourage it, in a controlled and purposeful way. Deploying your software into a production environment is not the end of the road, the job is really still only half done.
What are some of the types of testing that can be done in production? Well, let discuss a few.
Smoke Testing
Following any production release there needs to be a level of testing that is carried out. Often when using continuous delivery or deployment, there will be a set of automated smoke test that will run in the pipeline to verify the change and whether is has broken anything.
This isn’t always possible however depending on where the change has been made, so exploration testing again plays a huge part here. Both supplementing each other to learn more about the product that has been built and deployed.
This is often the most common type of testing in production. But what other types of testing can and should we be doing in production to maintain and measure quality?
Synthetic Testing (or monitoring)
Synthetic testing (or monitoring) is a really effective approach to monitoring and measuring quality in your production environments. These tests mimic real customer traffic (and journeys) by sending simulated requests to your applications and services from different browsers, devices and locations, and provide valuable insights both when they pass and fail.
As with all types of testing, there are benefits as well as challenges, which is why its important to supplement different types of testing with each other to get the full picture and context.
Some of the the benefits:
- Proactively identify performance problems
- Reduce MTTR (Mean time to resolution)
- Help meet application performance goals (SLO/A)
- Reduce risk of deploying code frequently
- Validate a product/service to a new market/location before customers access
Some of the challenges:
- Difficult to set up — test creation require coding skills
- Can be brittle — Small UI changes can create false alarms
- Lack context — when failures happen
Feature Flags
Another approach to testing in production is to use of feature flags (or toggle).
These provide a way to role out new features to only a subset % of customers at any one time. This allows the opportunity to gather insights as to how a new feature/product is performing and identifying any issues.
Essentially along with some of the testing already described above, the customer is actually helping test the new release in the real world. As only a small % of customers are seeing/using the new feature, if there is an issue the team have the option to fix forward, or disable the new feature by switching the feature flag off again.
As more and more confidence is gained around a new feature, the % of customers can be increased until eventually its at 100%. At this time, the feature flag can then be removed for the code base, and it is important they are because if left in the code base, over time that can all add up and have an impact on the complexity and performance of the code and applications.
A-B Testing (Experimentation)
Feature flags are also used to carry out A-B or multi-variant testing (Experimentations), which enable teams to have different design variants of a potential new feature on a website at the same time and balance those variants between the customers.
For example, if there are 2 variants — A and B — then a % of customers will see variant A and the other % will see variant B. These experiments are time boxed and the insights gathered are then measured to determine which variant performs better. These results can then guide the teams as to which design they would like to go ahead an build and roll out.
Performance Testing and Monitoring
Performance testing in a production environment is often where people raise big concerns, and to some degree, they are justified concerns. Concerns such as “what if we impact customers?” or “what if we take the application down and can’t recover quickly?”
That said, if you are able to do this in a controlled and structured way, you can gain some extremely valuable insights by carrying out performance testing in production.
As an example, at Dunelm we are moving towards a goal of ‘always being peak ready’. If the pandemic showed us anything, it was that you never know whats round the corner. During that time, when our stores were shut, our website was the only sales channel we had. This meant the traffic to the site was higher than we could have ever imagined….and we had no idea if it would handle it. This highlighted a great opportunity, and guided us to a different way of thinking and that is….peak can literally happen anytime, so we need to be always ready.
So, for example, rather than rushing around performance testing our Dunelm.com estate a couple of months before our normal peak times hit, with limited time to resolve issues, we now do this continuously throughout the year, in production, every 2 weeks.
By doing this, we are able to leverage the stable, larger production environment and monitoring, to gain insights around things that only occur under specific load that we may not have been able to see as clearly in the smaller, non production environments.
This means teams can work on improving these things and when they occur, rather than trying to unpick months of code changes, which is what would have needed to happened when only doing this kind of testing nearer the peak time.
Currently these test are triggered and monitored manually during the test runs, but the plan is to automate this and build in fail safes to kill the test to protect the website should there be an issue.
There are more types of testing you can do in production and the key thing to remember is that they all have their benefits and challenges, and ideally you want them to supplement each other to gain higher quality insights to learn from.
So other than testing in production, what else can we do in production that helps I gain more insights and learn about the product we build and the customers that use them?
Monitoring and Observability
In my last article I spoke about how important it was to ensure we build and implement the right monitoring and observability capabilities, and if we do that, then we can use them to help continue to build quality into our product and platforms, and prevent future issues from even happening.
Strong monitoring capabilities can help alert us to issues that are occurring there and then or will occur, whilst strong observational capabilities can help us debug issues to then fix and prevent them from happening again….and even proactively preventing future issues even happening in the first place.
So it’s really important that we build these up front and give our teams the best proactive visibility as possible.
What are the key benefits shifting right can bring to a team and an organisation?
When done well, the key benefits of shifting right are described below:
- Continuous Learning
- Higher Confidence
- Increased Customer Satisfaction
- Competitive Advantage
Bringing it together: Leveraging The DevOps culture
There is one common theme across these articles and that is continuous collaboration and feedback, and this is achieved through people, culture and mindset leveraging tools and automation. It’s really important to remember they are not independent of each other and to achieve continuous quality you need to focus on both together, and this is often a big cultural shift.
I hope you have enjoyed part 2 of this article focusing on shifting right and found it useful. Watch this space for some follow up articles delving more deeply into these different areas :)
