The Kernel Is a Trust Boundary

Everything in the OS Architecture and System Calls topics built one claim: user programs cannot touch hardware or each other’s memory directly; every privileged action goes through a syscall the kernel verifies. That makes the kernel the security boundary of the machine. An attacker who can trick the kernel (a syscall bug, an exploited driver, a TOCTOU race) has beaten every access-control policy above it.

This topic looks at the OS through that lens: how access decisions are made, how failures compound, and the concrete mechanisms — DAC/MAC, ACLs, capabilities, sandboxes, secure boot — that harden the boundary. The Security category deepens the network and application side; here the focus is the OS-level model.

Subjects, Objects, and the Access Decision

The classic access-control model has three parts:

• Subject — the actor: a user, a process, a thread.
• Object — the thing protected: a file, a device, a memory region, a syscall.
• Policy — who may do what to what, and how that’s decided.

Every syscall the OS performs is an access decision against this model: the kernel checks whether the subject (the calling process, running as some user) is allowed the requested operation on the object (the file, the socket). How that check is made distinguishes the two families of policy.

DAC vs MAC

• Discretionary Access Control (DAC) — the owner of an object decides who may access it. POSIX mode bits (rwxr-xr--), owners/groups, and ACLs are DAC. Flexible and familiar, but the owner can grant access to anyone — so a process running as you can read anything you can.
• Mandatory Access Control (MAC) — a system-wide policy the owner cannot override. Linux SELinux and AppArmor, and Windows Mandatory Integrity Levels (the “Low/Medium/High” tokens), enforce system-defined rules on top of DAC. Even if a process runs as root, MAC can still forbid the action — which is why MAC contains blast radius: a compromised nginx running as root on a MAC system still can’t read /etc/shadow unless the policy allows it.

The rule to internalize: DAC says “who owns it decides”; MAC says “the system decides, no exceptions.” Production servers harden with a MAC layer precisely because DAC can’t stop a compromised privileged process.

ACLs vs Capabilities

Beyond the coarse owner/group/other triple:

• Access Control Lists (ACLs) — per-subject exceptions attached to an object: “Alice can read, Bob can read/write.” POSIX ACLs (getfacl/setfacl), Windows NTFS ACLs. Fine-grained, but they scale poorly (one list per object, evaluated per access).
• Capabilities — instead of “who are you?”, ask “what are you allowed to do?”. A capability is an unforgeable token granting a specific right — open a socket below port 1024, bind a low port, change file ownership. Linux capability sets (cap_net_bind_service, cap_sys_admin, …) let a process drop privileges it doesn’t need. Containers lean on this: a container image runs with an explicit capability set, not full root.

Capabilities invert the mental model usefully: least privilege becomes “grant the minimum set of tokens,” which is far easier to audit than “trust this user.”

Privilege Separation

The single most important OS security pattern: split a program so that the risky part runs with the least privilege needed. The classic cases:

• OpenSSH splits into a privileged monitor (authenticates, then drops privileges) and an unprivileged slave that handles the session. A bug in the session code never runs as root.
• Web servers run a root master that binds port 80, then spawn worker children as an unprivileged user.
• Browsers (Chrome, Firefox) sandbox renderer processes so a parsing bug can’t read the user’s files.

The pattern is the same one you saw in Process Management (processes for isolation) applied deliberately: do the dangerous work in the least-privileged context you can.

Sandboxing and Syscall Filtering

A sandbox restricts a process beyond user/permission checks:

• seccomp (Linux) — a syscall allow-list; the kernel kills (or signals) the process if it calls anything not on the list. Docker’s default profile and Chromium’s sandbox both build on it.
• Capability dropping — setcap/prctl to shed capabilities early in startup, before untrusted input is parsed.
• Windows equivalents — AppContainer isolation and job objects restrict access tokens and resources for sandboxed apps; the same idea as seccomp + cgroups.
• Namespaces/cgroups (from Virtualization & Containers) — constrain what the process can see and use.

The combination — restricted syscalls + dropped capabilities + bounded resources + its own namespace — is how a container becomes a sandbox and how a renderer process becomes a sandbox. Each layer is one more thing an attacker must chain together.

TOCTOU: The Race That Beats the Policy

Time-of-check to time-of-use (TOCTOU) races are a class of OS bugs where the access decision and the operation don’t happen atomically:

1. An attacker arranges for the check to pass on one file.
2. Before the operation uses it, the attacker swaps in a different file (via a symlink or rename) that the check would have rejected.

The classic victim is a setuid program that checks “can I write to /tmp/log?” and then opens it — if a symlink to /etc/shadow replaces it between the check and the open, the program follows the link. The fixes are structural: open the file once and check the opened handle (openat/fstat), use atomic operations, or operate on file descriptors rather than paths. TOCTOU explains why kernel and privileged code must treat path resolution and operation as one atomic act.

Secure Boot and the TPM

The boot path is the first trust decision. Secure Boot (UEFI) verifies the signature of the bootloader, which verifies the kernel, which verifies loaded modules and drivers — building a chain of trust from firmware up. The Trusted Platform Module (TPM) stores keys and measures boot components (platform configuration registers) so software can prove to a remote party (attestation) that it booted known-good code. The connection to the OS: if the kernel is trusted, the syscalls it enforces can be trusted — Secure Boot protects the very boundary everything else depends on.

The OS Attack Surface

Worked Example: Least-Privilege Web Server

Trace why a hardened server runs the way it does:

1. nginx starts as root — only to bind port 80 (a privileged port) and read its config.
2. It drops capabilities: setcap cap_net_bind_service lets it bind without root, then workers run as user nginx.
3. MAC (SELinux/AppArmor) confines nginx to a domain that can read only /var/www and its config — not /etc/shadow.
4. Each worker’s seccomp profile blocks syscalls the server never needs (e.g., ptrace, module loading).
5. If any worker is compromised, it can read only what nginx can read, call only what seccomp allows, and see only its own namespace — every escalation path beyond is blocked or loudly logged.

Each layer is cheap on its own; together they make a single bug far from a full compromise.

Practice Trajectory

1. On Linux, compare ps -o user,comm -p 1 with the user of a browser’s renderer processes; identify the privilege-separation boundary.
2. Run capsh --print (or getpcaps <pid>) and enumerate the capabilities a process holds; drop the obvious extras.
3. Create a symlink attack against a temp file in a small C program and demonstrate the TOCTOU window; fix it with openat.
4. Inspect the default seccomp profile of a container (docker inspect); remove a syscall and watch a program fail.
5. Check whether Secure Boot is enabled on your machine and name the boot components it verifies.

When It’s the Right Tool