Why Testing in Production is Great…To a Point
A story of Elon Musk, public beta, and rethinking testing best practices.
Last week, Elon Musk rolled back the latest version of Tesla’s Full Self-Driving (FSD) public beta software.
Any new software deals with its fair share of bugs during development. But what shocked me the most was this sentence from Musk’s tweet following the announcement: “It is impossible to test all hardware configs in all conditions with internal QA.”
This tweet caused quite an uproar, from regulators who are questioning the safety of FSD technology to people that opted into trying the new software.
But Musk’s remarks, and especially the last sentence of his remarks, also sparked a lot of discussion in the QA and software testing community.
One particularly engaging discussion on my LinkedIn feed brought up many different concerns with the Tesla software rollback. Not only was the necessity of public beta called into the question, but also the entire notion of testing in production and when to use it.
Testing in Production: Is It Necessary?
In short, yes.
Testing in production adds a lot of value to the software development process. It is very difficult to do 100% testing before an application is deployed. New, untested features are often rushed into the production build before getting tested thoroughly.
While these last-minute, untested features might not be mission-critical to that particular release, they are still open to bugs and other problems. And for a truly great user experience, even bugs within less critical features can have adverse effects on the software quality.
By testing in production, teams can proactively address these untested features. Using application and real user monitoring, testers can detect and fix bugs within these features after the software has already been released. No matter at what stage of the development process, quality doesn’t get left behind.
Features that are tested in production can also be rolled out gradually, exposed only to specific user segments. There are two benefits to testing in production gradually:
- Early users have the opportunity to influence upcoming releases.
- Developers get “free” user feedback from a real production environment.
The better question to ask is, does testing in production work with a public beta?
Knowing Your Limits: When to Test in Production
While testing in production can be useful, there is a limit to its benefits, especially when testing in a public beta. There are two factors to keep in mind when considering testing in production: who is testing in production, and on which type of software.
Who Should Test in Production
While discussing the rollback on my LinkedIn feed, Michael Bolton echoed my opinions exactly when thinking about risk and production testing:
“It can be useful and okay to perform experiments in production, too, providing that the risk of harm is low and well-managed.”
Testing in production can be great, but only when the features being tested at this stage are low risk. One great way to manage risk while in public beta is by having developers test in production to preview the software at this stage.
A good example of a successful public beta is iOS15 and Android 12. Both were in beta for a few months, which allowed Apple and Google to increase the software quality for their upcoming releases.
Selenium 4.0 was also in beta for a long time. Appium 2.0 is still in public beta.
These software releases all benefitted from earlier releases in similar ways, such as getting feedback and shaping the future of their products. These software products have minor risks and are delivered using opt-in methods. Plus, the people who volunteered to participate in the beta did not implicate external users who had nothing to do with the product.
With a greater knowledge of the code behind the software, developers are the best type of persona to detect bugs after a public beta release. Using managed crowds to test a limited beta release is also a valid option.
But when it comes to Tesla’s public beta, releasing their latest software version to the public for testing poses way too high of a risk. Tesla’s safety scores that they are using to monitor those who opted into the beta are insufficient from a testing standpoint. These safety scores do not do nearly enough to minimize the risk of harm (or worse) to the Tesla driver, as well as any passengers or even nearby pedestrians.
This leads to the next factor to consider when testing in production: which type of software to test.
Which Type of Software to Test in Production
Another way to qualify whether it’s right to test in production is by considering the software itself, and whether it is part of an innately high-risk industry.
The automotive industry is not the place to test in production. With life-risking software, it is frankly a reckless and irresponsible move. Testing for such software should be more rigorous and only leave specific areas for public beta testing.
As my colleague Brijesh Deb of Infosys noted:
“Public beta for something with life at risk is a bad idea IMHO. This kind of an approach suits Facebook and not Tesla.”
Even when considering the recent Facebook, Instagram, and WhatsApp outage, the consequences of the bug that caused the outage were surely not life-threatening (although it certainly revealed how much we rely on social media). While Facebook certainly suffered from the outage, it did not have to contend with issues of physical harm or loss of life.
Chris Hyde of VividCloud pointed out in my LinkedIn discussion that testing mission-critical software in public beta stems from aggressive release pushes from the Product team. While a public beta may speed up releases, using one for this purpose also shows a lack of testing maturity.
Hyde continued:
“Mature teams…ensure that the features released to [beta and alpha] channels have appropriate diligence.”
Especially in industries where the lives of those using the software and others surrounding those users literally hang in the balance, testing maturity is essential.
Bottom Line: Test in Production, but in the Right Context
Mature testing is all about coverage, analytics, risk management, and responsibility. While testing in production offers valuable insights and can contribute to a mature testing model, it can only do so in the proper context.
Testing in production should only be done for lower-risk software that is not a central component of the application. Teams should also only test in production after coming up with a comprehensive testing strategy and executing it properly.
Even in the case of Tesla, there were other ways to mitigate risk with their 10.3 beta version. They could have specified that those who opted into the beta only test in specific areas, such as a racetrack. Industries that deal with life-and-death circumstances need to be even more careful about testing in production, setting up rigorous protocols to avoid fatal consequences.
But instead, the 10.3 FSD software beta serves as a cautionary example of irresponsible testing and shines a light on the limits of testing in production.
Enjoyed the article? Follow me on Medium to stay up to date on all things test automation and application development.
Join FAUN: Website 💻|Podcast 🎙️|Twitter 🐦|Facebook 👥|Instagram 📷|Facebook Group 🗣️|Linkedin Group 💬| Slack 📱|Cloud Native News 📰|More.
If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇
