Skip to main content
CAP theorem, consensus, message queues, microservices, and system design at scale.

Distributed Systems

CAP theorem, consensus, message queues, microservices, and system design at scale.

Microservices Architecture & Patterns

Monolith vs Microservices: An Economic Decision

A microservice is a small, independently deployable service owning its own data and its own lifecycle. The monolith — one deployable unit owning everything — is not a failure mode; it’s often the correct starting point. Microservices solve a real problem (independent scaling/deployment by team), but they tax everything: you pay distributed-system costs (network, consistency, observability, deployment) for every feature, not just the ones that need to scale.

The honest rule: start monolithic; decompose when a boundary pays for itself — a team that needs independent deploy cadence, a workload that needs different scale, a latency isolation boundary. Teams don’t fail because their monolith is too big; they fail because they microservice-ify without the operational maturity to pay the distributed tax.

Decomposition: Domain-Driven Design

The hardest part is finding the right service boundaries. The tool is Domain-Driven Design (DDD):

  • Bounded contexts — each service owns a bounded context: a domain area with its own model, language, and data. “Orders” in the sales context and “orders” in the shipping context are different things with different shapes.
  • Aggregates — a cluster of objects treated as one unit; the aggregate’s root is the only entry point. A service should own whole aggregates so cross-aggregate calls stay inside it.
  • Context mapping — explicit contracts between contexts (shared kernel, customer/supplier).

The practical test: if a “transaction” must span multiple services’ databases, you’ve likely drawn the boundary wrong — the data that changes together should live together.

Communication: Sync vs Async

  • Synchronous (HTTP/REST, gRPC) — request/response, easy to reason about, tight coupling to availability: the caller waits on the callee’s uptime and latency. A chain of sync calls multiplies latency and failure modes.
  • Asynchronous (message queues/events) — fire and forget, decoupled, buffer against bursts — but you lose the immediate response and must handle eventual consistency and dedup.

The resilient pattern is event-driven for the long tail, sync only where a response is truly needed (and protected by timeouts). Services communicate their state changes as events; consumers react. This turns “service B is down” from a hard failure into “B’s events are pending in the queue.”

The API Gateway and BFF

  • API Gateway — the single entry point: authentication, routing, rate limiting, aggregation. Clients don’t know the internal topology; the gateway does. Risk: a new bottleneck/single point of failure — keep it thin, don’t put business logic in it.
  • Backend-for-Frontend (BFF) — a per-client-type gateway (mobile BFF, web BFF) so each client gets an API shaped for it, without leaking the mobile-optimized payloads to the web. The pattern exists because one gateway that serves all clients becomes a lowest-common-denominator mess.

Service Discovery

Services move (deploys, autoscaling, failures) — nobody can hardcode IPs. Discovery answers “where is service X right now?”:

  • Client-side — clients query a registry (Consul, etcd) and pick an instance.
  • Server-side — a load balancer / Kubernetes Service resolves and forwards (DNS-based discovery: service-name.namespace.svc).

Kubernetes makes this native: a Service is a stable VIP over a set of pods, DNS-resolvable; the control plane (backed by etcd) does the discovery. Consul/ZooKeeper fill the same role outside Kubernetes.

Resilience Patterns

Distributed calls fail in new ways; the patterns are the shield:

  • Timeouts — every remote call needs one. No timeout = the caller inherits the callee’s failure indefinitely.
  • Retries with backoff — retry transient failures (jittered exponential backoff), but bounded, and beware the retry storm: all clients retrying a dead service at once. Cap concurrency and add jitter.
  • Circuit breaker — after N consecutive failures, open the circuit: fail fast (or serve a fallback) instead of hammering a dying service. Close it after a cooldown probes. Prevents cascading failure.
  • Bulkhead — isolate failure domains: separate thread pools/connection pools per dependency, so one slow dependency can’t consume all capacity (like a ship’s watertight compartments).
  • Fallbacks — return a cached/stale/default answer when the real call fails.

Together these are the difference between “a dependency hiccup” and “an outage.”

Observability Is Non-Negotiable

You can’t debug a request that spans 15 services by looking at one log file. You need:

  • Distributed tracing — a trace ID flows through every hop (OpenTelemetry); you see the request’s full path and per-hop latency.
  • Structured logs with the trace/request ID correlated.
  • Metrics per service (RED: Rate/Errors/Duration).

A microservice without tracing is a black box; a fleet of them without tracing is a mystery box. This is covered deeply in the DevOps observability topic — but it’s a design requirement of microservices, not an afterthought.

Practice Trajectory

  1. For an e-commerce system, draw DDD bounded contexts and decide which “orders” data lives where.
  2. Redesign a 3-service sync chain as events; identify where the response is genuinely needed vs where events suffice.
  3. Place an API gateway + a mobile BFF in an architecture and list what each handles.
  4. Wire a circuit breaker around a flaky dependency and trace the open/close/half-open transitions.
  5. Run a multi-service request and trace it end-to-end with OpenTelemetry; correlate logs by trace ID.

When It’s the Right Tool

SituationTakeaway
Team-scale independent deploysMicroservices pay off
Small team, early productMonolith first — decompose later
Service must own its data fullyDDD bounded contexts + aggregates
Response needed immediatelySync with timeout + circuit breaker
“Just tell other services”Events through a message queue
Debugging a 15-hop requestDistributed tracing, or you’re flying blind