Chaos engineering is the discipline of breaking systems on purpose to discover failure modes before customers do. It is not the discipline of randomly turning off servers, despite its common caricature. The mature practice is hypothesis-driven: state what should happen when a dependency fails, run an experiment that triggers the failure, observe whether the hypothesis holds, and feed the gap back into design.

The motivation: a service that has never been observed failing is not a reliable service. It is a service whose failure modes have not yet been discovered.

The Hypothesis-Driven Method

Four steps constitute a chaos experiment:

1. Steady-state — define the normal behaviour: latency p99 < 300ms, error rate < 0.1%, throughput > 1000 RPS. This is the metric the experiment will try to perturb.
2. Hypothesis — “If I terminate 30% of the payment-service’s pods, the steady-state metric will remain inside the SLO because the load balancer has redundancy and the degraded-resolution path will return a 503, not a hang”.
3. Injection — deliberately trigger the failure: kill 30% of the pods in production (or, more conservatively, in a staging mirror) and observe.
4. Conclude — if the hypothesis holds, file the experiment as confirmation and increase the blast radius next time. If it fails, abort the experiment, then treat the gap as a bug — the system broke under a failure mode the design should have covered.

Two non-negotiables: always have an abort condition, and always run during business hours. A chaos experiment run overnight, with no one to abort, is the worst-case-nightmare — an incident of your own making.

Failure Injection Taxonomy

The failure space is broader than just “kill a pod”:

The class of injection should match the class of risk you’re hypothesising about. If you’re worried about long-tail latency, inject network delay, not pod kills. If you’re worried about regional resilience, inject a region blackhole.

Blast Radius and Steady State

Two design axes for experiments:

The discipline of doubling the blast radius after each successful experiment surfaces the resilience ceiling gradually. The first time a hypothesis fails is the bug you fix; subsequent experiments probe adjacent failure modes.

Mature chaos programs run continuous experiments (every deployment, intentional small failures) and periodic larger-blast-radius experiments (quarterly, ambitious). The continuous experiments are alarm bells; the periodic are stress tests.

Game Day Structure

A game day is a scheduled exercise where the team runs experiments deliberately, observes outcomes, and writes up findings. A typical structure:

1. Pick a hypothesis — start with one the team is uncertain about, not one everyone is confident will hold.
2. Pre-game briefing — review steady-state metrics; declare abort conditions; assign observer roles; ensure on-call hasn’t been distracted.
3. Run — inject the failure; observe metrics; assert whether the hypothesis holds.
4. Post-game — within 24 hours, produce a written report: hypothesis, observed outcome, gap, design fix. If the hypothesis failed, an executable item goes to the team’s backlog.

A game day is the closest a team gets to deliberate, low-stakes practice for the high-stakes incidents that customers will eventually trigger. Teams that game-day quarterly go into real incidents already practiced.

Resilience Patterns vs Chaos Experiments

Two complementary disciplines:

The patterns are the design; chaos is the verification that the design actually held. Each pattern in isolation has well-known failure modes (timeouts without backpressure, retries without jitter, circuit breakers that never re-open); chaos experiments surface the gap between intent and reality.

Practice Trajectory

1. List five “this should just work” beliefs about your system — for example, “if the cache goes down, we fall through to the database”. These are your first five experiment hypotheses.
2. Pick one experiment and run it in staging: kill the cache pod; measure the steady-state metrics during the fallback. Identify whether the hypothesis held; fix the gap if not.
3. Schedule a quarterly game day with your team. Pick a hypothesis about an AZ failure. Document the abort conditions and the post-game write-up template.
4. Audit one resilience pattern in your code — say, retries. Jitter the backoff; rerun with and without jitter in a stress test, observe the difference in herd-retry behaviour.
5. Take an incident post-mortem from last year. Reconstruct the failure as a chaos experiment; run it in staging. Did the system recover the same way? Are the mitigations from the post-mortem still in place and observable?

When It’s the Right Tool