The db-caching topic covers caching atoms — cache-aside vs write-through, LRU vs FIFO, eviction policies. At scale, the questions change: not whether to cache, but where the cache sits, what happens when many caches miss together, and how invalidation is achieved when the cache lives outside your data center.

This topic is the distributed-systems angle on the cache.

The Multi-Level Hierarchy

A modern request crosses at least four cache layers between client and database — each with different cost, freshness, and invalidation mechanics:

The cost penalty per layer is ~10×. The user-visible stack is mostly: serve the same cached object from the closest available layer. The cheapest cache hit is one that never hits your origin.

Cache Stampede / Thundering Herd

A cache miss is automatic: if the cached value is missing, compute it and refill the cache. The disaster is when N clients miss simultaneously — a cold cache, an expired TTL, or an eviction under memory pressure — and they all race to compute and refill the same value.

       TTL expires
            ↓
   client1 → miss → compute → refreshing
   client2 → miss → compute → refreshing   (duplicate work)
   client3 → miss → compute → refreshing   (triple work)
   ...
   N simultaneous loads of origin — origin overloaded

Each duplicate load stresses the origin; the cache miss becomes a copy of the underlying expensive operation, multiplied by concurrency. Three production fixes:

1. Locks / Single-Flight

A coalesce: one request goes through to origin; the others wait. The Redis primitives SETNX or a per-key lock can implement this. Consequences:

• One origin load instead of N — the cost penalty is bounded by 1× origin work.
• The waiters must time out, or they hold the client request open forever.
• The lock-holder’s failure must trigger a retry by one waiter — that’s the production wrinkle of the pattern. (redlock, RedisLock libraries handle this.)

2. Probabilistic Early Refresh (XFetch)

A clever algorithm from Vattani et al.: when a client reads a cached value that is near its TTL, it probabilistically decides to refresh early — with probability proportional to remaining TTL. The result: most refreshes happen before expiry, smoothly distributed; the cache miss-on-expiry pattern never develops.

remaining = (ttl - now) / ttl            # in [0,1]
beta = 1                                 # tunable; >1 means refresh later
p = exp(-beta * beta * log(remaining))   # probability of refresh on this read
if (random() < p) refresh()

The good property: concurrent refreshes spread out over time rather than clustering at a single expiration instant. Production-grade caches (Redis with XFetch extensions, Snapchat’s Caching framework, Twitter’s CacheMetricsManager) ship variants of this.

3. Bloom-Filters / Negative Caches

When the computation is “does this key exist in the underlying store?” — and the answer is often no (e.g., “does this user-id have a notification?”) — a Bloom filter at the cache layer rejects the negatives. Only when the Bloom filter says “maybe” does the request fall through to the upstream.

The negative cache and the Bloom filter are complementary, not competing — the Bloom filter answers “in this set?”, the negative cache answers “not in this set recently.” Combined, they can eliminate most cold-key calls.

CDN Caching — Invalidation as a Hard Problem

The CDN cache lives outside your infrastructure perimeter; you cannot issue a del to Akamai or Cloudflare the way you do to Redis. The invalidation strategies form an explicit trade matrix:

The CDN choice is rarely “one of the above”; mature platforms combine TTL for the long tail, cache-tag purges for important updates, versioned URLs for static assets, and small TTL for things that change rapidly but aren’t worth purging explicitly.

Circuit Breakers Around Cache Failures

A cache layer failure (Redis cluster down) must not take down the application. Three architectural moves:

• Fail-open vs fail-closed — if the cache is down, the application falls through to the slower origin (fail-open) by default. Fail-closed (refuse calls while the cache is unavailable) is right only when the cache holds safety-critical data — rare.
• Bulkheads — separate thread pools / connections for cache vs origin paths, so cache slowness does not starve the origin path.
• Circuit breaker — after N consecutive cache timeouts, treat the cache as down for K seconds; do not bolt latency onto every request by waiting for it.

The deeper architectural truth: a cache layer unavailability should lower latency, not raise it. If your cache’s unavailability causes overall latency to go up — because every request now trips the timeout before falling through — your cache integration is wrong.

Practice Trajectory

1. Pick a hot request path in your system. Trace the four cache layers it crosses and the freshness guarantee each provides. Identify which layer is the cost ceiling.
2. Simulate a 0.1s TTL cache miss on 1,000 concurrent requests — observe the thundering herd. Add a single-flight lock and re-run. Add a probabilistic early refresh and re-run; compare the load distributions.
3. For an endpoint with lookup-by-id semantics, sketch a Bloom filter at the gateway. Estimate the false-positive rate at 1M IDs in 4MB of memory.
4. Identify a CDN-cached object in your system. Which invalidation strategy does it use? Which would be better?
5. Take your last cache-related incident. Was it missing cache, stale cache, or cache-as-source-of-truth? The fix depends on which one.

When It’s the Right Tool