A sharded Postgres with a coordinator is “horizontally-scalable SQL” only up to a point: each node is serialisable only against its own shards; cross-shard transactions areCoordinator-mediated and slow; consistent global secondary indexes are a myth. NewSQL — distributed SQL — emerged in the 2010s as the proposition: “Postgres semantics, global scale, strong consistency, linearisable writes.” Spanner, CockroachDB, TiDB, and FoundationDB are the production outcomes.

This topic is the architecture of distributed SQL: what it buys, what it costs, and when to use it.

Why “Global SQL” Is Hard

A SQL database promises strong ACID isolation. Doing that on one machine is the textbook problem; doing it across machines that can fail independently and maintain consistent state is one of the genuinely-hard research problems of the last twenty years.

The complications layer:

The architectural move of NewSQL is to commit each on its own, with one crucial addition: a globally-agreed time oracle that gives every write a unique timestamp ordering.

Spanner and TrueTime

Google’s Spanner (2012) is the origin. The deep insight: if you can bound the uncertainty of time globally — now ∈ [now_earliest, now_latest] with an error bound ε ≈ 7ms — you can assign timestamps that do not interleave between concurrent writes from different data centers. Distributed serialisability collapses to “wait out the uncertainty, then commit”.

The architectural primitives:

• TrueTime API — TT.now() returns an interval [earliest, latest]; TT.after(t) returns true if t has definitely passed; TT.before(t) if it’s definitely in the future. Implemented with GPS + atomic clocks in each data center to bound ε.
• Paxos per shard — each shard’s writes are committed via Paxos (a majority must agree). The Paxos leader assigns a timestamp.
• Commit-wait — when a write is committed at time t, the system waits until TT.after(t) is true before ack-ing to the client. This guarantees no future write can come in with a smaller timestamp that violates ordering.
• 2-safe replication — the Paxos group spans multiple data centers, with the leader in one; commit takes one cross-DC round trip.

The result: a globally-distributed SQL database with externally-consistent serialisable transactions, at the price of write latency = max(one cross-DC round trip, TrueTime uncertainty).

CockroachDB and TiDB

CockroachDB (Cockroach Labs, 2015) and TiDB (PingCAP, 2016) are the open-source descendants of Spanner’s architecture, both modeled around the same core ideas.

CockroachDB:

• Language: Go; storage engine: RocksDB / Pebble.
• Raft per range — each range (64MB shard of a table) is a Raft group.
• Hybrid Logical Clocks (HLC) instead of TrueTime — CockroachDB cannot assume GPS-corrected atomic clocks in every DC; HLC combines a physical clock with a Lamport-clock component to preserve causality despite drift.
• Read replicas for latency — read-only transactions can read from local replicas, accepting slightly-stale reads for non-strong reads.
• Geo-partitioning — the user can pin specific ranges to specific regions, controlling where a row’s Raft leader lives.

TiDB:

• Language: Rust + Go; storage engine: TiKV (Rust, RocksDB-based).
• Separation of storage and compute — TiKV is the distributed KV layer; TiDB nodes are stateless SQL layers; placement driver coordinates.
• Perf-dependent on SSDs — designed for NVMe-backed nodes; not typically deployed on spinning rust.
• Strong ecosystem for OLAP — TiDB + TiFlash (columnar replica) makes TiDB hybrid transactional/analytical (HTAP) out of the box.

Calvin and Deterministic Transactions

Calvin (Yale, 2012) is an alternative architecturally: instead of expensive 2PC and clocks, pre-determine the transaction order before they execute. All transactions enter a deterministic log; replicas run them in the same order; no locks needed because the order is decided up front.

• Pros: deterministic execution means replicas never need to coordinate beyond log replication; throughput is high.
• Cons: all transactions must be declared up front — including their read/write set — which precludes interactive transactions; the model fits workloads with predetermined transaction shapes (e.g., payments) but not ad-hoc SQL.

Calvin’s lineage lives on in FoundationDB’s transaction layer — FDB uses deterministic ordering on a single sequencer per database, with multi-region sequencers for high availability.

Linearisable vs Serializable

Two guarantees that NewSQL systems provide and that “good enough” distributed stores often don’t:

Spanner achieves strict serialisability via TrueTime and commit-wait. CockroachDB and TiDB achieve close variants with HLC / TSO. The reason the distinction matters: a database can be serialisable but not linearisable (“eventual consistency across rows, but transactions appear ordered”) — which is fine for analytics, ambiguous for the “did this write happen” question.

When to Use Distributed SQL

Distributed SQL has high operational complexity and per-write cost. Choose it deliberately.

The honest take: NewSQL is the right tool for a small fraction of applications. The default for most teams is still a single Postgres at moderate scale (~1TB, ~10K TPS), with sharding as a deliberate next step when scaling forces it.

Practice Trajectory

1. Pick a workload you operate. List three properties of it that would push you toward distributed SQL and three that would push you away.
2. Compare TrueTime and HLC: in two regions with 50ms network round-trip, what are the achievable latency properties? Which technique is right without atomic clocks?
3. Trace a single-row write through Spanner through the commit-wait model. Identify where the TrueTime uncertainty interval lives in the latency budget.
4. Configure CockroachDB or TiDB locally with three-node or single-node; execute a multi-statement transaction; forcefully kill one node mid-way; verify the database’s commit protocol handles it correctly.
5. Convert a small schema from Postgres to CockroachDB or TiDB. Note what changes (GeoPartition, replica placement); note what doesn’t (SQL semantics).

When It’s the Right Tool