Achieving Continuous Delivery

Continuous delivery (CD) is the opposite of what I call the all-nighter deployment cycle (ADC). The ADC is a recurring high-stress operation which involves large builds that often move after hours with “all hands on deck” and many hours of planning. Inevitably something is missed in the large feature set and your team spends the next few days putting out fires.

CD aligns with the agile manifesto in that it focuses on delivering small increments and adapting to change over following a plan. Features move to prod whenever the need is there rather than being bundled and pushed on a cycle or schedule. On the surface CD is like a magic trick – I write a few lines of code, and hours later they’re in a production environment making a positive impact in somebody’s life.

Few companies are able to achieve CD and therefore lack agility at the core – making it a real challenge to benefit from an agile methodology. You’ll never see exponential gain by simply wrapping agile around an app that depends on ADC.

So how do you achieve CD? It requires the following technical and cultural practices and capabilities:

Technical

A flexible branching strategy gives you the ability to push discrete features through testing environments and out the door while other development is actively taking place. Recommended approach: GitFlow

For very large features that really can’t be decomposed, feature flags allow you to push code to prod even when a feature isn’t ready to be consumed by your users.

Build server – You won’t get far without this! Ideally you have 3 types of jobs running on your build server

  • Continuous Integration (CI) – Builds app and runs all unit tests whenever a commit is detected
  • Nightly – run integration, E2E, deploy to lower environment
  • On demand – deploy to environments

Recommended server – Jenkins, or anything that works and you don’t have to pay for.

Automated Tests – They’re critical. It’s simply not feasible to push features out to Prod 3 times a day if your app takes days or even hours to manually regression test. As a general rule your QA resources should only be testing the changes that are moving, along with maybe a few sanity checks. Everything else is covered by one of the following tests:

Unit tests validate that a function does what it was designed to do. You don’t have to do full blown TDD, but if you never write tests you’re building a minefield around yourself that will be set off at the most inopportune time. I ask the guys on my team to commit at least one test for every new feature they add, and before fixing a bug to write a broken test that the bug fix will fix.

Integration tests validate interoperability with 3rd parties and test a potentially large scope of functions.

E2E tests use a headless client to actually simulate a user going through the app, and verify different results. Selenium coupled with nUnit is a great E2E approach for web apps.

SOLID Application Architecture – Should be simple, easy to understand, and able to handle change without breaking. This is achieved by following SOLID design principals, building cohesive classes and loosely coupled components.

Automated Monitoring and Logging

Read my post on Building a DevOps Dashboard

A bug that took a day for a user to report and a day to fix really has a 2 day turn-around time. By detecting the bug immediately you’ve cut the time to fix it in half.

Always-on infrastructure – Load balanced or hot-swappable. You should be able to deploy at any time without causing an outage. When was the last time you went to Facebook and saw a banner that said “down for maintenance”?

Automated rollbacks – You should be able to quickly revert to the last working release by running an automated job.

Cultural

Passion – Everyone has to think the product is the coolest thing ever and be stoked to get new stuff out the door. Accomplish this by giving your team true ownership.

Agile leadership mindset – leaders must be able to translate strategic objectives into a set of small, individually shippable features

Get rid of the fear – don’t let flashbacks from your ADC past derail you! Deploying doesn’t have to be scary. Fear and paranoia is one of the biggest hurdles to CD.