System Calls & Kernel Interface

The System Call Interface

The system call is the OS’s API. User-space programs cannot touch hardware, another process’s memory, or the page table directly — the CPU enforces this in user mode. Any privileged operation must go through a syscall: a controlled trap into the kernel that runs privileged code on the program’s behalf and returns.

This boundary is the entire reason the OS is trustworthy: everything a process does that matters goes through code the kernel verifies. When the OS Architecture topic traced a read() call, it showed the flow — here we go deeper into the interface itself.

How a Syscall Works

1. The program places arguments in registers (and a syscall number in a dedicated register — e.g., rax on x86-64).
2. It executes the special instruction syscall (x86-64) or svc (ARM).
3. The CPU traps to kernel mode, switching the stack to the kernel stack and jumping to the syscall handler.
4. The kernel validates arguments (pointers, permissions), performs the operation, and sets a result.
5. The handler returns to user mode with the syscall number’s return value in a register.

On x86-64 the syscall numbers are a fixed ABI table: 0 = read, 1 = write, 2 = open, 57 = fork, 59 = execve, and so on. This ABI is a contract — it must never change, which is why adding a syscall is additive (new numbers) rather than a renumbering.

Syscall vs Library Call

A library call (read() in glibc, printf, malloc) may do all its work in user space or wrap one or more syscalls:

This layering is why tracing library calls (ltrace) and tracing syscalls (strace) show different pictures, and why buffered I/O is so much faster than unbuffered: each syscall costs a mode switch, and buffering amortizes that cost across many logical operations.

The Cost of a System Call

A syscall is expensive relative to a function call because the CPU must:

• switch privilege levels (kernel/user) and re-validate state,
• switch stacks,
• flush or save parts of the pipelined CPU state,
• and (historically) flush the TLB.

A single syscall costs on the order of ~100 ns to ~1 µs — thousands of times a plain function call. At scale this dominates: a high-throughput server issuing one syscall per small I/O operation can be syscall-bound. This is the deep reason behind buffering, mmap, sendfile, io_uring (async, batched syscalls), and the general engineering rule: batch your system calls.

POSIX: The Portable Interface

POSIX is the standard that normalizes the syscall API across Unix-like systems (Linux, BSD, macOS). Code written against POSIX — open, read, fork, pthread_create — is portable to any POSIX system because those names map to each OS’s syscall ABI. The main POSIX categories:

The abstraction layer above POSIX (the VFS for files, sockets for networking) is why the same read() works on a file, a socket, and a device — the kernel funnels them through one interface.

Error Handling Convention

Syscalls report failure by returning -1 and setting the global errno. There is no exception mechanism — every call must be checked. -EINTR (interrupted by a signal) and -EAGAIN (would block) are not fatal errors but conditions to retry, and mishandling them is a classic source of subtle bugs. This “every call returns an error” style is the ancestor of Go’s explicit error returns and Rust’s Result.

Tracing with strace

strace intercepts every syscall a process makes, printing arguments and results:

strace ls
strace -e trace=open,read ls            # filter
strace -p 1234                          # attach to a running process
strace -c ls                            # syscall summary (count + time)

strace -c is the fastest way to answer “why is this slow?”: it shows which syscalls dominate, exposing pathological patterns like a syscall per byte, repeated stat calls, or constant connect retries. It is the first tool for the Debugging & Profiling topic, and the mental model of “every program is a stream of syscalls” underpins everything from performance engineering to container security.

Practice Trajectory

1. Run strace -c ls and name the top three syscalls; explain why read/write/stat dominate.
2. Compare strace -c for a buffered vs unbuffered C program writing 10 MB — watch the syscall count collapse with buffering.
3. strace a server you run and identify a single hot syscall to optimize (e.g., replacing read+write with sendfile or io_uring).
4. Use grep -E "syscall|nr=" /usr/include/asm/unistd_64.h to find how the ABI table is declared, and confirm read is entry 0.
5. Trace a program that handles EINTR: interrupt a blocking read with kill -INT and observe the retry loop.

When It’s the Right Tool