Why Cross-Service Transactions Are Hard

ACID works because one database has one transaction log and one lock manager. A distributed transaction spans multiple services, each with its own database — and no global transaction manager can atomically commit across independent engines and networks. The core problem is the coordination problem you already met in consensus: two databases can’t be told “commit” simultaneously and guaranteed to both succeed.

So the industry converges on two answers: two-phase commit (the old, heavyweight, rarely-used one) and the saga pattern (the practical, widely-used one).

Two-Phase Commit (2PC)

2PC adds a coordinator:

1. Prepare phase — the coordinator asks each participant to prepare: can you commit? Each writes its changes and a prepare record to its own log (now durably able to commit), and replies “prepared.”
2. Commit phase — if all reply prepared, the coordinator tells everyone to commit; if any replies no (or times out), everyone rolls back.

The guarantee: once a participant says “prepared,” it must be able to commit — so a coordinator crash between phases leaves everyone in doubt (blocked). Recovery requires the coordinator’s log (or a heuristic decision) to resolve. That blocking failure mode — plus the coordinator as a single point of failure and the fact that participants hold locks during the whole dance — is why 2PC is avoided in modern distributed systems (it only appears in XA transactions, and mostly in legacy or single-vendor stacks). It favors consistency over availability during failures, in direct tension with scale.

The Saga Pattern

A saga is a distributed transaction decomposed into a sequence of local transactions, each committed independently, with compensating transactions to undo the earlier ones if a later step fails.

• Order service creates the order (local commit).
• → Payment service charges the card (local commit).
• → Inventory service reserves stock (local commit).
• → If inventory fails: payment compensation refunds the charge; order compensation cancels the order.

Because each step commits locally, there’s no global lock and no coordinator blocking — the price is that intermediate states are visible to other consumers (the order exists briefly without a reservation), and recovery is a workflow, not a rollback.

Choreography vs Orchestration

Reality is a blend: an orchestrator for the critical path, events for fan-out. The orchestrator must be able to resume after a crash — its state is a workflow in a database, not process memory.

Idempotency: The Saga’s Safety Net

Compensations and retries both multiply operations, so every saga step must be idempotent:

• The client sends an idempotency key; the service stores the result per key and returns it on retry instead of re-executing.
• Compensations must be idempotent too — refunding an already-refunded charge must be a no-op.

Without idempotency, one network retry double-charges a customer. With it, the “at-least-once + dedupe” pattern makes sagas safe against the network’s lying (remember the fundamentals topic).

The Transactional Outbox Pattern

The gap in every event-driven saga: “update my database AND publish an event” is a local transaction + a side effect — if the publish fails after the DB commit, other services never hear about the change. The outbox pattern makes it atomic:

1. In the same local transaction, the service writes its domain change and an outbox row (the event, marked pending).
2. A poller (or the DB’s own replication mechanism — e.g., Postgres logical decoding, Debezium) reads new outbox rows and publishes them to the queue, then marks them published.

Now the event publication is derived from the committed local transaction — no “DB committed but event lost” window. Combined with idempotent consumers, this is the backbone of reliable event-driven systems (it’s how you make a queue almost exactly-once).

Choosing Your Trade

The recurring theme: strong consistency across services is expensive and rare; sagas + idempotency + outboxes give you reliability without global coordination. If a business invariant truly requires cross-service atomicity, the correct move is usually to move the data into one service/database rather than to build a distributed transaction around it.

Practice Trajectory

1. Decompose “order → payment → inventory → ship” into a saga and list the compensating transaction for each step.
2. Trace the failure at step 3 (inventory) and show the compensation workflow for both choreography and orchestration.
3. Add idempotency keys to payment and refund endpoints; verify a retried charge doesn’t double-bill.
4. Implement an outbox: write domain row + outbox row in one transaction, then a poller publishes and marks done.
5. Justify, for a real feature, whether the data should be in one service (ACID) or split (saga) — and defend it.

When It’s the Right Tool