SLI, SLO, SLA: The Reliability Contract

Reliability is a number with a target, not a vibe:

• SLI (Service Level Indicator) — the measured metric: availability (successful requests / total), latency (p99 < 300ms), durability, throughput.
• SLO (Service Level Objective) — the target you commit to internally: “99.9% of requests succeed over 30 days.” The product is measured against this.
• SLA (Service Level Agreement) — the external, contractual commitment to customers — with penalties. SLAs are business documents; SLOs are engineering budgets.

Crucially: an SLO is a promise you intend to occasionally break. “99.9%” means 0.1% of the time you are allowed to be slow or down — and that allowance is the point.

Error Budgets: The Freedom to Ship

The error budget is the allowed failure: 100% − SLO. At 99.9%, the budget is 43 minutes of downtime per month. This turns reliability into a budget that engineering actually spends:

• If the service is over budget (too many errors), stop risky changes — feature freezes until reliability recovers. Reliability gets prioritized because it must.
• If the service is under budget (very healthy), spend the budget: ship the risky-but-valuable change, do the big migration, experiment.

This is the radical idea: the error budget authorizes risk-taking. Without it, either nothing ships (everyone fears breaking the magic number) or everything ships recklessly (nobody knows the number). The budget makes the trade explicit and governed by data, not vibes.

Alerting That Doesn’t Burn People Out

The on-call rule: pages are for when a human is needed now. Everything else is a dashboard or a ticket.

Alert on SLO burn, not on every anomaly. A service hitting 99.9% SLO with occasional 5-minute hiccups is healthy; paging on every hiccup destroys the on-call’s nights and trains them to ignore the pager. On-call fatigue is a reliability failure mode exactly like a crash.

Incident Response: Roles and Flow

When the page fires, the response is a process, not heroics:

1. Triage / mitigate first — the first goal is restoring service (roll back, roll forward, scale, cut traffic), not root-causing. “Mitigate before you understand.”
2. Declare severity — SEV1 (service down / data loss / customer impact), SEV2 (degraded), SEV3 (minor). Severity drives who gets paged and whether an incident channel opens.
3. Roles — Incident Commander (owns the response, delegates tasks), Communications (posts status updates — “when did it start, what’s affected, ETA”), Operations (the mitigators). One person drives; everyone else executes tasks. No parallel uncoordinated fix attempts.
4. Log everything — the timeline is the raw material for the postmortem.
5. Resolve → review — restore, verify, then run the postmortem regardless of outcome.

Runbooks are the multiplier: a pre-written playbook (“if 5xx > 5% for 10 min, check X, restart Y, roll back Z”) turns a confused firefight into a checklist execution. Every alert links to one.

Blameless Postmortems

After every incident (even small ones), the team writes a blameless postmortem:

• A timeline — what happened, in order, from signals to resolution.
• The contributing factors — technical and process (a bad deploy, a missing canary, a confused runbook, a race).
• Action items — concrete, owned, dated improvements that prevent the class of incident, not just this one.

Blameless is the non-negotiable rule: the postmortem is not about who made a mistake but about what in the system allowed the mistake to hurt customers. If people fear blame, they hide incidents and the learnings vanish. The best postmortems are written about your own “dumb” mistakes — they’re the highest-leverage engineering artifacts in the company.

Capacity and Load: The Proactive Side

Reliability isn’t only reacting. Capacity planning (modeling growth against headroom), load testing (finding the breaking point before the flash crowd does), and game days (rehearsing failover, killing a node on purpose) find the failure modes that alerts can’t — because they happen before anything is broken.

Practice Trajectory

1. Define an SLI, SLO, and error budget for a service; compute the monthly allowed downtime.
2. Decide the page-vs-ticket threshold for an error-rate metric and defend it in SLO terms.
3. Run a mock SEV1 with an incident commander and a runbook; time from page to mitigated.
4. Write a blameless postmortem for a real (or invented) incident with timeline + action items.
5. Load-test a service to find its saturation point and feed the number into a capacity plan.

When It’s the Right Tool