Debugging & Profiling

Debugging Is a Method, Not a Tool

The difference between a junior and a senior engineer isn’t knowing more tools — it’s a repeatable method. The scientific workflow applies everywhere:

1. Form a hypothesis about the cause (not the symptom).
2. Design an observation that would confirm or refute it (a trace, a profile, a reproducer).
3. Run it.
4. Update the model, repeat — never “fix” blind.

Most long debugging sessions fail because people jump straight to step 3 with no hypothesis, or fix a symptom instead of the cause. Start every investigation by reproducing the problem, because a problem you can reproduce is a problem you can solve.

Start at the Boundary: strace

The Syscall topic introduced strace; here it becomes a first-line tool. Because every interaction with the outside world is a syscall, strace shows you exactly what a process is doing, not what it claims to do:

strace -f -p 1234                 # follow threads on a live process
strace -c myprogram               # syscall summary — where does time go?
strace -e trace=network curl -v https://example.com

Classic wins: a program that “hangs” is actually blocked on a read() of a socket that never sends data; a “slow” program is doing one syscall per byte; a “file not found” is actually a permission check failing deeper (open with -EACCES). Look at the error returns — strace prints them, and the negative results are usually the bug.

Interactive Debugging: gdb

For a crashing or misbehaving program, gdb gives you the machine’s point of view:

gdb ./server core               # debug a core dump
gdb --args ./server -c prod.cfg # debug a live run with args
(gdb) run
(gdb) bt                       # backtrace — the call stack at crash
(gdb) frame 3                  # inspect a specific frame
(gdb) print variable           # examine values
(gdb) break open               # breakpoint on a function
(gdb) info registers           # CPU state

The two highest-value commands are bt (where did it crash — which function, what arguments, what callers) and inspecting local variables at the failing frame. This is where understanding the stack from the systems-programming topic pays off: a backtrace is literally the call stack of stack frames.

Core Dumps

When a process crashes, the kernel can write a core dump — a snapshot of the process’s memory at the moment of death. You debug it like a paused program: gdb ./binary core. A core file is effectively the answer to “what happened?” preserved in a file, which is why production systems configure core dumps (with limits!) and why crash-reporting services (and Rust’s panic + coredump tooling) are built on this mechanism. Enable them deliberately; an unconfigured system silently discards the best forensic evidence.

Profiling: Where Does the Time Go?

Debugging fixes bugs; profiling finds the performance problem. The two dominant techniques:

• Sampling CPU profiler (perf record + perf report, or perf top) — periodically samples the instruction pointer; the aggregate shows which functions consume CPU. Cheap overhead, good enough signal.
• Flamegraph — a visual rendering of profiler output: each bar is a function call stack, width is time, so the widest towers are your hot path. Read it top-down; the horizontal widest spans are what to optimize.

perf top                      # live hot functions (like top, for code)
perf record -p 1234 sleep 5   # sample a process
perf report                   # interactive report

A CPU flamegraph answers CPU time. For I/O-bound or blocked processes, the hot path is invisible to the CPU sampler — that’s when you combine it with strace -c (syscall time) and memory profiling.

Memory Profiling

Leaks and bloat are memory problems, not CPU problems. Tools: valgrind (full heap analysis — slow, for correctness like leaks/uninitialized reads), heaptrack/massif (allocation timelines), and gdb’s heap inspection. In managed runtimes (Go, JVM, Node) the runtime exposes heap snapshots and live objects. The pattern to hunt: memory grows without bound → keep a heap profile history → find what’s referenced but never freed, or what’s cached without eviction.

Common Production Failure Patterns

A cheat sheet of what the tools typically reveal:

Sanitizers (compile with -fsanitize=address,thread) are the highest-leverage debugging tool: they turn silent memory corruption and data races into loud, precise crashes during development.

Practice Trajectory

1. Reproduce a “hang”: start a server, strace -p it, and identify the exact blocking syscall.
2. Write a small C program that dereferences a freed pointer; compile with ASan and watch it pinpoint the line.
3. perf top a busy process (e.g., a grep -r over a big tree) and identify its hot function.
4. Generate a flamegraph for that process and name the widest call stack.
5. Take a crashing program, enable core dumps, and answer “where and why did it crash?” from gdb core alone.

When It’s the Right Tool