Why Linux
Practically the entire systems engineering stack — cloud VMs, containers, Kubernetes nodes, CI runners, database servers — runs on Linux. This topic connects the OS theory (processes, syscalls, file systems) to the concrete environment: where things live, how to control processes and services, and how to inspect a running system. If the Shell & Command-Line topic taught you the language, this one teaches the country you’re operating in.
The Filesystem Hierarchy
Linux follows the Filesystem Hierarchy Standard (FHS) — a canonical layout every admin and every container expects:
| Path | Contents |
|---|---|
/bin, /sbin | Essential executables (now symlinks to /usr/bin) |
/etc | System configuration (text files — the classic Unix config model) |
/home, /root | User home directories |
/var | Variable data: /var/log logs, /var/lib state, /var/tmp |
/tmp | World-writable scratch (cleared on boot) |
/proc, /sys | Virtual file systems exposing kernel state |
/dev | Device files |
/run | Runtime state since boot |
In containers, this hierarchy is the image: a Docker image is essentially a chroot’ed FHS tree, which is why understanding the layout transfers directly to container debugging.
Users, Groups, and Permissions
Linux is multi-user by design. Every process runs as a user; every file has an owner, a group, and a 9-bit mode (rwxr-xr--). Two mechanisms go beyond basic chmod:
- Setuid (
chmod u+s) — a program runs with the owner’s privileges regardless of caller. This is howsudoandpasswdwork, and it’s a top attack target (privilege escalation). - ACLs — per-user/group exceptions beyond owner/group/other (
getfacl/setfacl).
sudo is the disciplined way to escalate: a specific command as a specific user with full audit trail, instead of running everything as root. Running as root by default is the single most common cause of catastrophic rm -rf and config mistakes.
Processes and Signals
From the process-management topic: a process is created by fork+exec, tracked in a PCB, and ended by exit. Operationally:
ps aux # snapshot of processes
top / htop # live view
pgrep / pkill -f pattern # find/kill by name or args
kill -TERM <pid> # polite termination
kill -KILL <pid> # forced, uncatchable
nohup cmd & # survive shell logout
Signals are the OS-level notifications: SIGTERM (graceful shutdown), SIGKILL (force), SIGINT (Ctrl+C), SIGHUP (terminal closed / reload for daemons), SIGSTOP/SIGCONT (pause/resume). “Graceful shutdown” in production (draining connections, flushing state) is a process choosing to handle SIGTERM — which is why container orchestration sends SIGTERM, waits, then escalates to SIGKILL.
systemd and Service Management
systemd is the modern init system: PID 1, the first process the kernel starts, responsible for launching and supervising everything else. Services are defined by units (/etc/systemd/system/*.service):
systemctl status nginx # is it running? show logs
systemctl start/stop/restart nginx
systemctl enable nginx # start at boot
systemctl daemon-reload # re-read unit files after editing
journalctl -u nginx -f # follow the service's logs
A unit declares ExecStart, Restart=on-failure, dependencies (After=network.target), and resource limits. systemd supervising processes is the same supervisor/supervisee pattern as containers and orchestrators — learn it here and the container world is familiar.
Package Management
Distribution packages are how software gets installed reproducibly:
- Debian/Ubuntu:
apt update && apt install nginx(.debfiles) - RHEL/Fedora:
dnf install nginx(.rpmfiles) - Alpine (containers):
apk add nginx
The package manager resolves dependencies, runs install scripts, and (critically) records what is installed at what version — the machine-level analog of the lockfiles from the Build Tools topic. Container base images are packages frozen into a filesystem layer.
Logs and /proc
Two places answer “what is happening?”:
- Logs —
/var/log/holds system and service logs;journalctlreads systemd’s structured journal. Structured logging (JSON, timestamps, levels) in modern apps is a direct descendant of this model — and the seed of the observability topic. /proc— a virtual file system that is kernel state:/proc/cpuinfo,/proc/meminfo,/proc/<pid>/(per-process:/proc/1234/status,/proc/1234/fd/,/proc/1234/environ). Reading/procis how monitoring tools (top,ps, container runtimes) gather their data.
Practice Trajectory
- Walk the FHS: identify where config, logs, and state live on your machine; check
/var/log,/etc, and/proc/meminfo. - Create a user, a group, and a file readable only by the group; verify with
ls -landid. - Launch a daemon-like process in the background, find it with
ps aux | grep,SIGSTOPit, resume it, thenSIGTERMit gracefully. - Write a systemd unit for a tiny script,
systemctl startit, and confirm it restarts on failure. - Install a package with your distro’s manager, then use
journalctland/var/logto see its logs.
When It’s the Right Tool
| Situation | Takeaway |
|---|---|
| Any server or VM | Linux fluency is the baseline requirement |
| Container debugging | Images are FHS trees; ps//proc/journalctl work inside them |
| Deployments | systemd units and package managers are the smallest IaC |
| Security | Users, setuid, and sudo discipline are the access-control model |