Pular para o conteúdo principal
CI/CD, containers, orchestration, infrastructure as code, cloud, and observability.

DevOps & Infrastructure

CI/CD, containers, orchestration, infrastructure as code, cloud, and observability.

Release Engineering: Canary, Blue/Green, Feature Flags

A deployment puts software into an environment. A release lets users see it. The two were the same action for a decade; the practices in this topic are about prying them apart. The motivation is blast radius: when release and deploy coincide, every mistake is a customer-facing incident; when they are separate, every mistake is a thing you observe on ten users before a hundred thousand see it.

This topic is what happens after the artefact has cleared CI.

Deployment vs Release

ConceptDefinitionFailure scope
DeployThe binary/container is running somewhere in the clusterInternal — no user impact
ReleaseUsers are routed to the new binaryExternal — user-facing blast radius
RollbackUsers routed back to the previous binaryOperational — minutes of degraded service
Roll forwardA new version is shipped that fixes the bug, no rollbackLongest repair window, but no degraded state in the interim

Conflating deploy and release forces every bad deploy to be a customer incident. Decoupling them is the foundational move underneath every strategy below.

Strategies by Blast Radius

StrategyPropertiesWhen it shinesWhen it hurts
Recreate (down then up)Old version terminated; new version launchedOnly when state is incompatible; otherwise unavailableThe whole system is down during the swap
Rolling updateN at a time, old replaced with newStateless services with N+1 capacitySlow; mixed-version window
Blue / greenTwo identical environments; flip routerPre-prod validation in the live environment2× resource cost during release
CanarySend 1% → 10% → 100% of trafficAnything with a measurable health signalRequires the observability to detect on small samples
A/B testTwo variants live side-by-side; users routedProduct decisions on user-visible behaviourSlowest; tied to product analytics, not infra health
Shadow / dark launchNew code receives real traffic but responses are discardedHigh-risk changes where correctness must be proven under loadInfra cost doubled; idempotency burden for side effects

The choice is by blast radius tolerance: how few users, for how long, with what observability. A B2B airline reservation system tolerates zero customer-facing exposure on the new version during business hours, so it relies on blue/green plus shadow. A consumer web app tolerates hours of partial exposure in exchange for fast learning, so it canaries.

Feature Flags as the Universal Primitive

A feature flag is an if (flag.enabled) { new } else { old } in the code path, where enabled is read at runtime from a configuration store. Flags turn “deploy the binary” into “ship the code; turn on the flag later, slowly.”

Flag kindLifetimeExample
Release flagDays to weeks“Roll out new checkout to 2% → 100%”
Ops flag (kill switch)Indefinite — always present until needed“Disable non-critical work during an incident”
Experiment flagWeeks“A/B test cart page layout”
Permission flagForever“Premium feature enabled for tenant X”

Three rules compound:

  1. Flags are debt — every flag is a code branch that has to stay coherent. Retire release flags within weeks; an ever-growing flag set is a slow-burning bug farm.
  2. Flag evaluation is a tier — a flag service (LaunchDarkly, Unleash, Flagsmith) needs an SLA; if it’s down, your service must default to a safe behaviour, not fall over.
  3. Flags enable decoupling — release on Tuesday, turn on Thursday; a bug is a flag=false, not a hotfix redeploy.

Progressive Delivery

Progressive delivery is canary + automation + metrics-driven promotion. The pipeline observes a health signal (error rate, latency p99, conversion rate) and auto-promotes the canary to a larger percentage — or auto-rolls back if the signal degrades.

Two production-grade tools:

ToolShapeWhere it lives
FlaggerKubernetes-native; pairs with any service mesh that exposes bool-stop signals (Istio, Linkerd, NGINX)Within the cluster; reads SLOs you’ve defined
Argo RolloutsKubernetes controller; canary as a CRD with steps (setWeight 5%, pause 2m, setWeight 20% …)Same shape; declarative-as-code releases

The insight both embed: the release is a YAML declaration, not a manual Friday-night ritual. The rollout’s success criteria are written as code; the operator is off-shift.

Rollback Semantics

“Rollback” is a four-syllable word that hides a per-system decision: what does rollback mean here?

System shapeWhat “rollback” must do
Stateless microserviceRe-route to previous ReplicaSet — seconds
Stateful service with backwards-compatible schemaRe-route to previous replica + leave new schema in place
Stateful service with schema migrationRollback to previous binary + forward-migrate the schema back; or accept a rollback script per migration
Event-sourced systemReplay events from a checkpoint; or kill the bad projection and re-derive
Database with destructive migrationRestore from backup + replay CDC; the slowest, the one you must avoid

Rollback is a property you design for, not a property you discover. A migration that is not reversible (drops a column, changes a column type) means rollback is impossible without data loss — which means you have to ship the change in three reversible steps (expand → migrate → contract) instead of one. The pattern is named expand/contract:

  1. Expand: add the new column / table / shape alongside. Both versions are valid.
  2. Migrate: copy and transform old → new.
  3. Cut over: switch reads/writes to the new shape.
  4. Contract: remove the old shape only after you’re confident you don’t need to roll back.

Rollback is now possible at every step before Contract — and Contract is a fully separate, deliberate release.

Practice Trajectory

  1. Pick a system you operate. Sketch the release matrix (recreate / rolling / blue-green / canary) and a recommended choice with justification.
  2. Add a release flag to one feature in your code. Ship the binary with the flag off. Turn it on for one tenant; observe; ramp to 100%. Describe what metrics justified the ramp.
  3. Write a Flagger or Argo Rollouts YAML for a canary with a 30-minute ramp, an SLO threshold of 0.1% error rate, and an automatic rollback.
  4. Pick a destructive schema migration you’ve shipped or would ship. Re-plan it as an expand/contract sequence of three migrations. Identify the step at which rollback becomes hard.
  5. Run a “rollback drill” against a recent release: trigger the alert, execute the rollback, time it. The drill tells you whether “rollback works” is hope or reality.

When It’s the Right Tool

SituationTakeaway
Mistakes happening at release timeDecouple deploy from release — feature flags
Small, observability-rich change on a high-traffic pathCanary with automated rollback
Incompatible schema or breaking state changeExpand/contract; never a single destructive migration
On-call rotation often needing rollbacksLeave clear, tested rollback instructions; treat them like runbooks not folklore
“We can’t ship until we can roll back”Correct — design the rollback path first; ship the second-best release style that survives the worst case