System Design: Trade-Offs Made Explicit

System design is the craft of turning vague requirements into a concrete architecture — and, more importantly, of naming the trade-offs you’re making and why. This topic is a practice ground: the workflow, the math, and four canonical designs that exercise every pattern in this curriculum.

The Workflow

A disciplined sequence beats cleverness every time:

1. Clarify requirements — functional (what must it do) and non-functional (QPS, latency, durability, scale). Ask the scale numbers first: they drive everything.
2. Estimate capacity — back-of-envelope: QPS × payload = bandwidth; storage = writes/day × size × retention; read/write ratio shapes the cache.
3. High-level design — the pieces (clients, load balancer, API, services, DB, cache, queue, CDN) and the data flow.
4. Deep-dive the hot paths — the write path, the read path, the one thing that actually matters at scale.
5. Failure modes & trade-off discussion — what breaks, and what you gave up to get the rest.

Numbers Everyone Should Know

Back-of-envelope math is the difference between a plausible design and nonsense. The canonical reference latencies (approximate, but right in magnitude):

Capacity arithmetic: a typical server handles ~10k–50k req/s for simple stateless work; one network RTT of 1 ms means a single-threaded request chain tops out at ~1000 req/s — concurrency is how you go faster. Storage: 500M rows × 200 bytes = 100 GB — fits in memory on a couple of nodes; 50B events/day × 1 KB = 50 TB/day — that’s a log store problem, not a transactional one. Always do the multiplication before choosing the architecture.

Worked Design 1: Rate Limiter

Goal: limit a client to N requests/sec without slowing legitimate traffic or blowing up memory.

• Data: per-client counters. In-memory (Redis) with the token bucket algorithm — a bucket refills at rate r and holds at most b tokens; each request takes one. Handles bursts (b) while enforcing the average (r).
• Placement: at the edge (API gateway / load balancer) so the limiter protects the whole fleet, not one instance. A Redis cluster holds counters; a local cache + periodic sync avoids a Redis hit per request.
• Response: 429 Too Many Requests + Retry-After. Put the limit in headers (X-RateLimit-Remaining) so clients can behave.
• Trade-offs: distributed counter sync costs consistency; sliding-window vs token-bucket trades burst tolerance for simplicity.

Worked Design 2: URL Shortener

Goal: tiny.com/abc123 → long URL, with redirects at web scale.

• Short code: base62 of 7 chars ≈ 62⁷ ≈ 3.5 trillion combos. Generate by counter + base62 encoding (not hash — no collisions to handle).
• Write path: POST long URL → allocate code, store code → long URL (+ created_at, expires_at).
• Read path: GET /code → look up → 302 redirect. This is the hot path (9:1 reads) — cache the mapping, put a CDN in front if traffic justifies.
• Storage: relational DB with the code as primary key (the code is the index). Shard by code hash if writes grow; the counter is the only shared state (a DB sequence or a dedicated counter service).
• Numbers: 100M URLs/day × 300 bytes ≈ 30 GB/day ≈ 11 TB/year — cheap at any tier. ~1200 writes/s, ~11k reads/s — one well-tuned DB + cache handles it.

Worked Design 3: Chat / Messaging

Goal: 1:1 and group chat, ordered, with presence and push.

• The core decision: ordering. Messages must be ordered per conversation. Give each conversation a sequence number (a per-conversation counter) — clients display by it, so ordering is deterministic even when delivery isn’t.
• Storage: one row per message in a table keyed (conversation_id, seq) — range reads per conversation are trivially indexed. Old messages go to cold storage.
• Delivery: the classic trade — WebSocket for live channels, REST fallback for polling when the socket drops, push notifications for mobile. Exactly-once isn’t achievable; at-least-once + client-side dedup (by message ID) is the honest contract.
• Fan-out: for large groups, publish once to the conversation, let consumers read their own offset — don’t copy the message to every member’s mailbox (a write amplification of N).

Worked Design 4: News Feed

Goal: each user sees a merged, ranked feed of posts from people they follow.

• Push (fan-out-on-write) — when a user posts, write the post into each follower’s feed (cached list). Reads are trivial; writes fan out by follower count. Great for small/medium follow counts.
• Pull (fan-out-on-read) — on read, fetch the followed users’ recent posts and merge/rank. Writes are trivial; reads pay for large follow sets.
• The real system is hybrid: push for the majority of “normal” follow counts, pull for celebrities with millions of followers. Ranking (recency × engagement score) happens at read time on the merged candidate set.

The Recurring Toolkit

Every design is the same ten tools in a new arrangement: load balancer → stateless API → cache → DB (+ queue for spikes, + CDN for static, + object storage for blobs, + sharding when one node saturates). If you can name, per component, what it does, when it’s needed, and what it costs, you can design any system — the canonical designs above are just practice at choosing.

Practice Trajectory

1. Estimate capacity for a 1B-user social feed: QPS, bandwidth, storage, cache working set.
2. Design a rate limiter with token bucket and justify Redis vs local counters.
3. Design a URL shortener to handle 10x traffic: where do you shard, and where does the counter live?
4. Design chat with per-conversation sequence numbers; explain how ordering survives a reconnect.
5. For a news feed, choose push vs pull vs hybrid for a user with 10 vs 1,000,000 followers.

When It’s the Right Tool