Observability: Questions You Didn’t Know to Ask

Monitoring tells you something is broken (you’ve defined the alert). Observability is the ability to ask new questions about a running system — “which DB is this slow query hitting?” — without deploying new code. The three pillars — metrics, logs, traces — are complementary lenses over the same events; observability is the discipline of making all three cheap and correlated.

The Three Pillars

None is sufficient alone. Metrics tell you the system is degraded; logs tell you which request; traces tell you where in the call graph the time went. A production issue is usually: metric alert → correlate by trace ID → read the relevant logs.

RED and USE: The Two Metric Lenses

• RED (service-level, request-centric) — for each service, track Rate (requests/s), Errors (error rate), Duration (latency distribution, p50/p95/p99). The classic “is this service healthy” dashboard.
• USE (resource-centric) — for each resource (CPU, disk, memory, network): Utilization (% busy), Saturation (how much queueing/contention), Errors (hardware/software errors). The classic “is this node running out of headroom” dashboard.

RED is how you see the user experience; USE is how you see the machine. Both, together, diagnose most incidents.

Latency Percentiles

Averages lie — a p99 of 2s with a p50 of 40ms is a terrible experience for 1% of users, invisible in “average latency.” Track p50/p95/p99 (and p99.9 for critical paths). The p99 is the “tail” — and because a distributed request’s latency is the max (not the sum) of its parallel hops, tail latency compounds: five sequential hops at p99 each put you at p99⁵ territory. “Tail latency at scale” is the reason you see so much cache/redundancy engineering.

Metrics Collection: Prometheus and Pull

Prometheus is the de-facto metrics system: services expose a /metrics endpoint, and Prometheus pulls samples on a scrape interval into a time-series DB. Query with PromQL:

rate(http_requests_total{status="5xx"}[5m])

Grafana renders dashboards and alerts. The pull model has a nice property: if a service is dead, its metrics stop arriving — absence is itself a signal. Keep metric cardinality bounded (label values that explode — user IDs as labels — will kill any time-series DB).

Structured Logging

Logs should be structured (JSON, with fields) and correlated (a request_id/trace_id on every line) — not prose. Structured logs are queryable; prose is only greppable. Every log line carries its request context so a filtered log view of one trace tells the whole story.

Collection: the app writes to stdout (a container convention), a collector (Fluent Bit, Vector, the OpenTelemetry collector) ships it to an aggregation store (Loki, Elasticsearch), and retention tiers keep recent logs hot and old logs archived. Logs are the highest-volume, highest-cost pillar — sample verbose logs, keep error paths complete.

Distributed Tracing: OpenTelemetry

When a request crosses services, each service’s logs are islands. Tracing instruments the request itself: an OpenTelemetry agent injects a trace_id + span_id header, each service records a span (name, duration, parent), and the collector assembles the trace — a tree of spans showing where the time went. This is what makes a 15-service request debuggable (you met this in Microservices).

The discipline: propagate the context (libraries do this if you configure them), add spans at meaningful boundaries (DB calls, external HTTP, queues), and sample wisely (head-based sampling for latency profiles; keep error traces always).

Alerting: The Discipline That Pays Rent

A metric without an alert is a dashboard; an alert that fires every day at 3pm is noise that trains people to ignore it. Alerting rules:

• Alert on user impact and SLOs (error rate, p99), not on every metric wiggle.
• Prefer consecutive windows (e.g., 15 min of 5xx > 1%) over single-spike alerts.
• Every alert needs a runbook link and an owner. An alert without a runbook is a stress test.
• Alertmanager deduplicates, groups, routes (by severity/team), and escalates. On-call fatigue from noisy alerts is the #1 cause of ignored pages — and ignored pages are how incidents become outages.

Practice Trajectory

1. Instrument a toy service with RED metrics (counters for rate/errors, histogram for duration) and scrape with Prometheus.
2. Build a Grafana dashboard with the p50/p95/p99 latency panel and a 5xx error-rate panel.
3. Add structured logging with request_id and query one request’s full lifecycle from the aggregation store.
4. Add OpenTelemetry tracing across two services and view one trace with all its spans and hop latencies.
5. Write an alert on the error rate with a 15-minute window, attach a runbook link, and trigger it to verify the page path.

When It’s the Right Tool