The Network Is Not Reliable

The moment you run your program on two machines, a new class of problem appears that simply doesn’t exist on one machine: the two machines can disagree about what happened. This topic is about the frame — the fallacies, failure models, and mental tools you need before any specific protocol (consensus, queues, replication) makes sense.

The Eight Fallacies of Distributed Computing

The classic list, given by Peter Deutsch and James Gosling, is worth memorizing — every one of them is a real incident:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn’t change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

The healthy reflex is to assume the opposite of each until proven otherwise: packets drop, latency spikes, bandwidth is precious, packet inspectors exist, nodes come and go, teams conflict, and your stack is heterogeneous.

Partial Failure: The Defining Difference

On one machine, a function call either returns or raises — the machine crashes and everything stops, which is easy to reason about. Across a network, a request can:

• arrive, be processed, and return;
• arrive, be processed, but the response is lost;
• never arrive;
• arrive twice (a retry you didn’t intend);
• arrive at a node that is overloaded and slow.

This is partial failure: part of the system succeeds while another part fails, and you cannot always distinguish “slow” from “dead.” Timeouts are a guess, not a fact. This uncertainty is the reason we need timeouts, retries, idempotency, and eventually consensus and quorums.

Time and Order: Physical vs Logical Clocks

Two nodes cannot be assumed to share a clock. NTP keeps them approximately aligned, but “approximately” is the enemy of correctness:

• Physical clocks can jump backward (a clock-sync correction), so you can’t use them to order events across machines.
• Logical clocks — Lamport clocks, vector clocks — capture what you can actually know across a network: causality. If event A caused event B, then A’s logical timestamp is less than B’s. If two events have no causal link, they are concurrent and may be observed in either order.

The rule: physical time for duration (timeouts, TTLs), logical time for causality (ordering, versioning).

Network Partitions and the CAP Triangle

A partition is the network cutting the cluster into groups that cannot talk to each other. When partitioned, a service must choose: keep serving (available, but two sides may serve contradictory state) or stop serving writes (consistent, so there is a single, unified answer when healing). This choice is the essence of the CAP theorem — covered in depth in its own topic. For now: partitions will happen, and your system must have a declared behavior when they do.

Idempotency: The Retry Safety Net

Since retries are mandatory (the network drops things), operations must be safe to repeat. An operation is idempotent if doing it twice has the same effect as doing it once:

• GET — naturally idempotent.
• DELETE /user/42 — idempotent (the second delete is a no-op).
• POST /pay — not idempotent: two posts can charge twice.

The fix is an idempotency key: the client sends a unique key (X-Idempotency-Key: abc123), and the server stores the result for that key — a retry with the same key returns the stored result instead of re-executing. Every retryable write path needs one.

The Distributed Mindset

A checklist for reasoning about any distributed component:

1. What fails? (network, node, disk, clock)
2. What does the client see? (partial success is the default)
3. What happens on retry? (idempotency?)
4. What happens on partition? (availability vs consistency decision)
5. How do we know it worked? (observability, but that’s later in the curriculum)

Everything that follows — consensus, message queues, replication, microservices — is an answer to one of these questions.

Practice Trajectory

1. For your last three “flaky” outages, classify them against the eight fallacies.
2. Draw the four partial-failure outcomes for one HTTP POST and design the retry + idempotency behavior for each.
3. Construct two events on different nodes that NTP clock skew could reorder, and show a logical clock ordering them correctly.
4. Simulate a partition in a two-node service and decide, for a shopping cart, whether you’d rather be available or consistent during it.
5. Add an idempotency key to a payment-like endpoint in a toy service and verify double-submission returns one charge.

When It’s the Right Tool