A deployment puts software into an environment. A release lets users see it. The two were the same action for a decade; the practices in this topic are about prying them apart. The motivation is blast radius: when release and deploy coincide, every mistake is a customer-facing incident; when they are separate, every mistake is a thing you observe on ten users before a hundred thousand see it.
This topic is what happens after the artefact has cleared CI.
Deployment vs Release
| Concept | Definition | Failure scope |
|---|---|---|
| Deploy | The binary/container is running somewhere in the cluster | Internal — no user impact |
| Release | Users are routed to the new binary | External — user-facing blast radius |
| Rollback | Users routed back to the previous binary | Operational — minutes of degraded service |
| Roll forward | A new version is shipped that fixes the bug, no rollback | Longest repair window, but no degraded state in the interim |
Conflating deploy and release forces every bad deploy to be a customer incident. Decoupling them is the foundational move underneath every strategy below.
Strategies by Blast Radius
| Strategy | Properties | When it shines | When it hurts |
|---|---|---|---|
| Recreate (down then up) | Old version terminated; new version launched | Only when state is incompatible; otherwise unavailable | The whole system is down during the swap |
| Rolling update | N at a time, old replaced with new | Stateless services with N+1 capacity | Slow; mixed-version window |
| Blue / green | Two identical environments; flip router | Pre-prod validation in the live environment | 2× resource cost during release |
| Canary | Send 1% → 10% → 100% of traffic | Anything with a measurable health signal | Requires the observability to detect on small samples |
| A/B test | Two variants live side-by-side; users routed | Product decisions on user-visible behaviour | Slowest; tied to product analytics, not infra health |
| Shadow / dark launch | New code receives real traffic but responses are discarded | High-risk changes where correctness must be proven under load | Infra cost doubled; idempotency burden for side effects |
The choice is by blast radius tolerance: how few users, for how long, with what observability. A B2B airline reservation system tolerates zero customer-facing exposure on the new version during business hours, so it relies on blue/green plus shadow. A consumer web app tolerates hours of partial exposure in exchange for fast learning, so it canaries.
Feature Flags as the Universal Primitive
A feature flag is an if (flag.enabled) { new } else { old } in the code path, where enabled is read at runtime from a configuration store. Flags turn “deploy the binary” into “ship the code; turn on the flag later, slowly.”
| Flag kind | Lifetime | Example |
|---|---|---|
| Release flag | Days to weeks | “Roll out new checkout to 2% → 100%” |
| Ops flag (kill switch) | Indefinite — always present until needed | “Disable non-critical work during an incident” |
| Experiment flag | Weeks | “A/B test cart page layout” |
| Permission flag | Forever | “Premium feature enabled for tenant X” |
Three rules compound:
- Flags are debt — every flag is a code branch that has to stay coherent. Retire release flags within weeks; an ever-growing flag set is a slow-burning bug farm.
- Flag evaluation is a tier — a flag service (LaunchDarkly, Unleash, Flagsmith) needs an SLA; if it’s down, your service must default to a safe behaviour, not fall over.
- Flags enable decoupling — release on Tuesday, turn on Thursday; a bug is a
flag=false, not a hotfix redeploy.
Progressive Delivery
Progressive delivery is canary + automation + metrics-driven promotion. The pipeline observes a health signal (error rate, latency p99, conversion rate) and auto-promotes the canary to a larger percentage — or auto-rolls back if the signal degrades.
Two production-grade tools:
| Tool | Shape | Where it lives |
|---|---|---|
| Flagger | Kubernetes-native; pairs with any service mesh that exposes bool-stop signals (Istio, Linkerd, NGINX) | Within the cluster; reads SLOs you’ve defined |
| Argo Rollouts | Kubernetes controller; canary as a CRD with steps (setWeight 5%, pause 2m, setWeight 20% …) | Same shape; declarative-as-code releases |
The insight both embed: the release is a YAML declaration, not a manual Friday-night ritual. The rollout’s success criteria are written as code; the operator is off-shift.
Rollback Semantics
“Rollback” is a four-syllable word that hides a per-system decision: what does rollback mean here?
| System shape | What “rollback” must do |
|---|---|
| Stateless microservice | Re-route to previous ReplicaSet — seconds |
| Stateful service with backwards-compatible schema | Re-route to previous replica + leave new schema in place |
| Stateful service with schema migration | Rollback to previous binary + forward-migrate the schema back; or accept a rollback script per migration |
| Event-sourced system | Replay events from a checkpoint; or kill the bad projection and re-derive |
| Database with destructive migration | Restore from backup + replay CDC; the slowest, the one you must avoid |
Rollback is a property you design for, not a property you discover. A migration that is not reversible (drops a column, changes a column type) means rollback is impossible without data loss — which means you have to ship the change in three reversible steps (expand → migrate → contract) instead of one. The pattern is named expand/contract:
- Expand: add the new column / table / shape alongside. Both versions are valid.
- Migrate: copy and transform old → new.
- Cut over: switch reads/writes to the new shape.
- Contract: remove the old shape only after you’re confident you don’t need to roll back.
Rollback is now possible at every step before Contract — and Contract is a fully separate, deliberate release.
Practice Trajectory
- Pick a system you operate. Sketch the release matrix (recreate / rolling / blue-green / canary) and a recommended choice with justification.
- Add a release flag to one feature in your code. Ship the binary with the flag off. Turn it on for one tenant; observe; ramp to 100%. Describe what metrics justified the ramp.
- Write a Flagger or Argo Rollouts YAML for a canary with a 30-minute ramp, an SLO threshold of 0.1% error rate, and an automatic rollback.
- Pick a destructive schema migration you’ve shipped or would ship. Re-plan it as an expand/contract sequence of three migrations. Identify the step at which rollback becomes hard.
- Run a “rollback drill” against a recent release: trigger the alert, execute the rollback, time it. The drill tells you whether “rollback works” is hope or reality.
When It’s the Right Tool
| Situation | Takeaway |
|---|---|
| Mistakes happening at release time | Decouple deploy from release — feature flags |
| Small, observability-rich change on a high-traffic path | Canary with automated rollback |
| Incompatible schema or breaking state change | Expand/contract; never a single destructive migration |
| On-call rotation often needing rollbacks | Leave clear, tested rollback instructions; treat them like runbooks not folklore |
| “We can’t ship until we can roll back” | Correct — design the rollback path first; ship the second-best release style that survives the worst case |