A service steady-state under nominal load is the floor of its design. Under overload, two services diverge: those with backpressure and load shedding stay partly usable; those without become fully unresponsive. The reason is never-limits-flatten to a single moment of cliff failure. The practices in this topic are the circuit breakers of distributed systems — the discipline that turns cliff into slope.

The Shape of Overload

A service has a capacity — QPS or concurrency — beyond which adding load degrades throughput. Two regimes:

   throughput
       ↑        .-----.-------  plateau (graceful)
       |       /       \
       |      /         \---- cliff (catastrophic)
       |     /
       +----/----------------→ offered load
            capacity

• Cliff is unmanaged queue growth: requests pile up in queues, threads contending on connection pools, GC pressure builds, latency explodes, timeouts cascade up the call tree. The system collapses.
• Plateau is managed: extra load is shed early (rejected) rather than stacked; the system holds its maximum throughput while serving requests it can actually complete.

The design target of every technique below is plateau, not cliff.

Little’s Law as the Ceiling

For any stable queueing system: L = λ · W — the average number in the system (L) equals the average arrival rate (λ) times the average wait time (W).

The deep consequence: if a service’s average latency at full utilisation is W = 50 ms, the maximum throughput sustainable is bounded by your concurrency: λ_max = L / W = concurrency / 50ms. You cannot exceed Little’s Law; you can only raise the ceiling by adding capacity (raising L) or lowering latency (lowering W).

When load exceeds λ_max, the only options are shed, queue, or degrade. A service that does none of these three is just cliff-bound.

Token Bucket and Leaky Bucket

Two classical rate-limiters:

Token bucket is the right primitive for rate-limiting a client (“at most 100 QPS, burst 20”). Leaky bucket is the right primitive for shaping traffic (“I want a steady 100 QPS to my downstream”). Both are individually-per-endpoint limits; in a system with many endpoints, per-endpoint capacity is not enough — a single bad client bursting on one endpoint can still starve others.

Adaptive Concurrency

Static limits (“this service allows 100 concurrent requests”) are wrong most of the time — under-provision at peak, over-provision at low load. Adaptive concurrency adjusts the limit at runtime, using observed latency as the load signal.

Two algorithms in production:

Both are the congestion-control heritage of TCP, applied to service call traffic. Implementations: Netflix’s concurrency-limiter, Adaptive Concurrency Limit in gRPC, Netflix Scylla; commonly paired with a request queue where excess calls wait.

The property adaptive concurrency buys: the system holds its plateau under arbitrary offered load because the limit tracks actual capacity, not a guess.

Timeout Budgets and Request Hedging

A request that traverses five services cannot have five independent timeouts;timeoutA = 1s, timeoutB = 1s, … can result in a 5-second downstream chain. The pattern is deadline propagation: the caller gives itself, say, a 2s budget; each downstream call subtracts how long it took and propagates the rest to the next hop. The deadline tightens through the chain.

Hedging is the technique that keeps p99 latency low without making p50 regression: most requests complete on the initial call; the slow tail is hedged automatically. The cost is doubling load on the slow tail — hedging must be paired with adaptive concurrency, or the hedged load collapses the downstream.

Priority Queues

Under overload, not all requests matter equally. A typical priority order:

The shape is admission control: protect critical traffic by refusing non-critical traffic. Doing this calls for per-priority queues at the front door (a per-priority weighted gate ahead of the request handler). Without per-priority admission, a flood of analytics writes can starve checkouts — a typical “the metrics pipeline took down production” incident.

Graceful Degradation as a Product Decision

The deepest fact: shedding load is a product decision, not a pure-infrastructure one. Refusing an analytics write under load is fine; refusing a checkout is not. The degradation strategy — what the system does when it cannot serve — has to be visible to and approved by product because the failure modes are user-experience questions.

Three patterns of graceful degradation:

1. Cache fallback: serve stale data when fresh data is unavailable. Stale for an additional 30 seconds is usually preferable to a 500 error.
2. Read-only mode: allow GETs, refuse writes. The system stays partly usable during recovery.
3. Feature shutdown: disable the specific feature whose backend is unhealthy (disable comments, keep checkout; disable personalised feed, keep generic feed).

Each is a choice the system operator cannot make alone.

Practice Trajectory

1. Take a service you operate. Compute its λ_max from Little’s Law at its current latency ceiling. Compare to a recent peak in the traffic graph — did you operate at or beyond capacity?
2. Add a token-bucket rate limiter at the front of the service. Pick b and r to match a typical user (not the worst user). Observe whether tail latency falls or rises.
3. Walk the request budget: pick a long-tail user request that traverses 5 services. Replace per-hop timeouts with a 2s deadline propagated across each hop. Compare the failure modes.
4. Set up adaptive concurrency (AIMD or Vegas) on a downstream call. Run a stress test ramp from 10 to 10,000 QPS. Observe the limit and observe plateau throughput.
5. Pick an overload scenario in your own system. Outline the priority order (P0–P3). Argue the product side of graceful degradation — what is acceptable to drop, what must survive, who approved it.

When It’s the Right Tool