Performance work has two halves: measurement and interpretation. Most engineering effort is spent on the former; the senior skill is the latter. The discipline that Brendan Gregg’s Systems Performance (2013, 2nd ed. 2020) assembled is a checklist against leaping to conclusions — a sequence of small, repeatable analyses that turn “the system is slow” into “the system is slow because /dev/sda1 is at 99% utilisation with a 40 ms queue, and the application is configured to read each file twice”.

This topic is the framework — not the tools’ man pages.

Why Averages Lie

A metric like “average CPU: 40%” tells you nothing. A CPU at 40% could be:

• Smoothly handling two cores of work with eight idle → low latency, great.
• One of eight cores at 100%, the rest idle → head-of-line blocking, p99 latency very high.
• Briefly spiking to 100% in 5ms bursts and otherwise idle → tail latency spikes invisible in the average.

Three corrections:

• Use percentiles, not averages — p50, p99, p99.9. A p99 of 1.5 seconds hidden in an average of 200 ms is the experience your users actually have.
• Distributions, not points — the histogram (Prometheus histogram, HDR) reveals a bimodal distribution where the average misleads.
• Decompose the metric — “CPU” is not one thing. It is user, system, iowait, softirq, steal. Each is a different problem.

USE for Resources, RED for Services

Two complementary checklists.

USE (Utilisation, Saturation, Errors) — for every resource (CPU, disk, NIC, queue):

RED (Rate, Errors, Duration) — for every service:

The expert move is to apply USE and RED simultaneously: USE tells you which resource is saturated; RED tells you what user-visible effect that saturation has. Both together distinguish “the disk is slow” (USE only) from “the disk is slow and the user p99 jumped from 200ms to 2s” (RED adds).

The Latency Cliff at Saturation

The most important interpretive insight: latency is flat until a resource saturates, then it climbs steeply. The curve:

latency
   ↑                              __________________
   |                            /
   |                          /
   |            flat_________/
   |                         
   +--------------------→ offered load
                   capacity

Below capacity, queue is zero and latency is the service time. Above capacity, the request waits — and queueing is non-linear: at 80% utilisation, average wait is ~4× service time; at 95%, ~20×; at 99%, ~100× (M/M/1 model). A 1% capacity gain above 95% utilisation is more valuable than a 30% gain in the flat zone.

This is why SRE practice obsesses over “headroom”: the cliff is steep enough that targeting 50–70% peak utilisation is the sane production choice. Aim for >90% sustained and the tail spins out of control.

Flame Graphs

A flame graph is a visualisation of stack-trace samples: width is time-in-stack, height is stack depth, the y-axis is the call stack.

        ┌──────────────────────────────────────────────┐
        │ main                                          │
        └────────┬──────────────────────────────────────┘
                 │ handle_request
                 ├────────────┬──────────────────────┐
                 │ parse_input │ serve_request
                 │             ├──────────┬────────┐
                 │             │ db_query  │ render
                 │             │           │  ┌──────┐
                 │             │           │  │format│
                 │             │           │  └──────┘
                 └─────────────┴───────────┴─────────┘

• Width is hot — wide bars are in-stack the longest; reducing them is the highest-leverage fix.
• Stack depth is the call chain — deep narrow stacks are deep recursion; wide flat stacks are “broadly expensive”.

Two flavours:

• On-CPU flame graph — samples while the CPU is executing the program. Shows where active CPU time goes.
• Off-CPU flame graph — samples where the program is blocked (I/O, lock, syscall). Shows where latency goes when CPU time isn’t the bottleneck.

Most performance bugs are either “wide bar in the wrong place” (on-CPU) or “off-CPU flame high in futex_wait” (synchronisation). Reading both against each other is the analysis skill.

perf and eBPF Tracing

perf is the Linux profiler; eBPF is the programmable tracer. Both let you ask the kernel what it is doing, without restarting anything.

• perf record -F 99 -g -- <command> — sample at 99 Hz, capture call graphs. Post-process with perf report or render flame graphs with Flamethrower / flamegraph.pl.
• perf stat — counts hardware counters: cycles, instructions, cache misses, branch misses. Tells you what category of cost your code incurs.
• bpftool, bpftrace — eBPF analytics. A script like bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }' shows which processes call openat() and how often, live in production, with negligible overhead.

eBPF’s raison d’être: observability without perturbation. The kernel lets you attach tiny sandboxed programs to almost any kernel hook — syscalls, network packets, scheduling events — and aggregate, without a kernel rebuild or a process restart.

The 60-Second Performance Checklist

Brendan Gregg’s “Linux 60-second performance analysis” — run this before you look at the code:

1. uptime — load averages, and how much higher than CPU count.
2. dmesg | tail — recent kernel errors (OOMs, disk I/O errors, link flaps).
3. vmstat 1 — CPU, memory, swap, I/O at one-second granularity.
4. mpstat -P ALL 1 — per-CPU breakdown (single-core saturation?)
5. pidstat 1 — per-process CPU.
6. iostat -xz 1 — per-disk utilisation, await, queue depth.
7. free -m — memory and cache state.
8. sar -n DEV 1 / sar -n TCP,ETCP 1 — NIC throughput, TCP retransmits.
9. top — for the rough summary.

This sequence identifies the bottleneck category (CPU, memory, I/O, network) within a minute. The rest of performance work is then targeted: flame graphs for CPU, biolatency for I/O, tcpdump for network, perf record for code attribution.

Practice Trajectory

1. On a Linux box, run the 60-second checklist while under load. Identify one anomaly and one sentence of interpretation.
2. Generate a flame graph for an application you own. Find the widest bar. Spend ten minutes tracing what would have to be true to shrink it.
3. Use bpftrace on sys_enter_openat while the application is doing something heavy. Find the top caller. (Once was usually the answer. Once was not.)
4. Take one of your service’s p99 latency measurements. Plot its distribution; is it bimodal? If yes, decompose by sub-latency (where does the variance come from?).
5. Pick a saturated resource in production. Apply the USE/RED overlay: which metric gives the cause (USE) and which the user-visible effect (RED)?

When It’s the Right Tool