Continuous Deployment (CD) is often seen as the “Holy Grail” of software development. A developer checks in code, and it is miraculously deployed and tested in the QA, Stage and Production environments, without needing any human intervention at all. This sounds great- and it is- but only if you are ready for it! Here are five reasons that your team might not be ready for Continuous Deployment.
Reason One: You Don’t Have Enough Test Coverage
Sometimes teams can be so excited to set up Continuous Deployment that they don’t pay attention to what they are testing. It’s great to have tests pass and deployments automatically go all the way to Production, but if you are missing tests for important functionality you’re going to need to remember to do manual testing with every deployment. Otherwise, something could break and the automated tests won’t pick up on the problem.
The remedy: Make sure you have all the tests you need before you set up CD.
Reason Two: Your Tests Are Flaky
If your tests aren’t reliable, you are going to get all sorts of false failures. With CD set up, this means that deployments will fail. If your developers are trying to deploy to the QA environment, but they can’t get their code there because of your flaky tests, they will be annoyed. And no one wants to have to stop what they are doing to investigate why your automation failed in Production.
The remedy: Make sure your tests are reliable. If there are flaky tests, pull them out of the test suite until you can fix them, and make sure that you are manually testing anything that’s no longer covered by automation.
Reason Three: Your Tests Take Too Long
UI tests can take a very long time. If you really want to set up CD, you’ll have to consider just how much time they are taking. If developer A checks in code which kicks off the tests, and then has to wait for an hour to find out if the tests have passed, and meanwhile developer B checks in code which now has to wait until the first deployment has completed, soon you will have a mess on your hands.
The remedy: Make sure your tests are fast. See which tests you can shorten through strategies like: switching to API tests for testing back-end logic, setting up your test data ahead of time, using API and other services calls to set up conditions for tests, running tests in parallel, and eliminating redundant tests.
Reason Four: You Don’t Understand the Deploy Process
Having CD set up won’t be helpful at all if you and your team don’t understand how it works. When things go wrong with a critical deployment, you don’t want to have to find someone in DevOps to help you diagnose the issue. That will waste the DevOps member’s time and cause stress for everyone on the team.
The remedy: Make sure everyone on the team understands the deploy process. Learn how to configure the deploys, what common errors mean, how to fix a hung deploy, and so on. Take turns monitoring the deploys and solving problems so you aren’t dependent on one team member who can then never take a vacation.
Reason Five: You Don’t Have Alerting Set Up
Just because your deployments are now automatic doesn’t mean you can sit back, relax, and never think of them again! Sometimes your tests will fail, sometimes your connections to dependencies won’t get configured properly, and sometimes a flaky thing will happen that will fail the deployment. You don’t want to find this out from your CEO, or someone in DevOps, or your customers!
The remedy: Make sure that you have alerting and paging set up when deployments fail. You could have the person that made the code change get paged when there’s a failure, or you could have everyone on the team take turns being the one on duty for that week. Make sure everyone takes their paging seriously; if they’re on call for a week where they’re going to be on vacation, they should find someone to substitute for them.
Continuous Deployment, when done correctly, is a valuable tool that makes it easier for teams to quickly produce quality software. But be sure that you are completely ready for this step by taking an honest, objective look at these five reasons with your team.
Pingback: Five Blogs – 14 December 2020 – 5blogs
Is hard to find the balance between enough and not enough tests (it is even possible to have enough tests? 😀 ). I think there is much more value in having some flaky tests and improving them than having no tests and during every release praying for no crashing app. And this way we smoothly pass from flakiness to well setup alerts 😉
I think it depends on where and when those flaky tests are running, Dave. If they’re running as a nightly job, flakiness isn’t a big deal. But if the flaky tests are running as part of a deployment, they’ll just wind up causing frustration to the team. No one will know whether the deployment failed because of a flaky test or because there’s really something wrong. So I’d suggest keeping those flaky tests out of the CD pipeline until they aren’t flaky.