Replication: Copies for Read Scale and Durability

Replication keeps copies of the same data on multiple nodes. It buys two things: durability (a copy survives a node loss) and read scale (many readers spread across replicas). The central question is who may write, and when do the others catch up?

Single-Leader Replication

One node is the leader (primary): it accepts writes, appends them to a replication log, and followers apply that log. Followers serve reads but reject writes.

• Synchronous replication — the leader waits for the follower to confirm before acknowledging the client. Loses nothing on a single failure, but a slow follower slows every write.
• Asynchronous replication — the leader acknowledges immediately and streams changes to followers in the background. Fast, but if the leader dies before a follower catches up, the un-replicated writes are lost.

The classic compromise: synchronous to one follower, async to the rest. That way at least one node is guaranteed to have the data.

Replication lag is the real-world headache: a client writes to the leader, reads from a follower that hasn’t caught up, and sees stale data. Real fixes are read-your-own-writes (route the user’s reads to the leader for a moment) and monotonic reads (a user always reads from the same replica).

Multi-Leader and Leaderless

• Multi-leader — several nodes accept writes and replicate to each other. Enables multi-region writes, but creates write conflicts that must be resolved (last-write-wins, merge, application-level). Conflicts are the price of letting more than one node accept writes.
• Leaderless (Dynamo-style) — any node accepts reads and writes; the client reads from several and uses versions to reconcile. This is where quorum arithmetic comes from: with N replicas, a W/R-quorum (W + R > N) guarantees the read sees the newest write if it waits for enough nodes.

Sharding: Splitting the Data Itself

Replication gives every node the same data. Sharding splits the data across nodes — each node owns a different shard (also called a partition). Scale the write path horizontally; the trade is that queries touching many shards become scatter-gather, and cross-shard transactions are expensive or impossible.

Sharding strategies:

Consistent hashing fixes the rebalancing problem of naive % N: when the number of shards changes, only 1/N of keys move instead of nearly all of them. Each shard owns an arc of a hash ring; nodes and keys hash onto the same ring, and a key belongs to the first node clockwise from it.

The Rebalancing Reality

Data grows, so shards must split and move. Moving data is bandwidth + a window of reduced redundancy, and it can cascade: shard X gets too big → it splits → traffic shifts → neighbor shards overload. Production systems watch shard size distribution continuously and rebalance incrementally. Tools (Cassandra’s virtual nodes, or a directory service) make the mechanics automatic — but the observability of skew is a team responsibility.

Practice Trajectory

1. Draw the single-leader flow for a write and trace when a follower’s read can be stale; pick the moment “read-your-writes” must apply.
2. Compare W/R quorums for N=3 (W=2,R=2 vs W=1,R=3) and reason about which is faster and which is safer.
3. Hash the keys user-1..user-10 with % 3 and with consistent hashing on a ring of 3 nodes; count keys moved when a 4th node joins.
4. Design a shard key for a chat app (write-heavy, one user’s messages must be co-located) and justify it.
5. Explain why a multi-leader setup in two regions eventually converges — and what conflict resolution policy you’d choose.

When It’s the Right Tool