The System Call Interface

The system call is the OS’s API. User-space programs cannot touch hardware, another process’s memory, or the page table directly — the CPU enforces this in user mode. Any privileged operation must go through a syscall: a controlled trap into the kernel that runs privileged code on the program’s behalf and returns.

This boundary is the entire reason the OS is trustworthy: everything a process does that matters goes through code the kernel verifies. When the OS Architecture topic traced a read() call, it showed the flow — here we go deeper into the interface itself.

How a Syscall Works

1. The program places arguments in registers (and a syscall number in a dedicated register — e.g., rax on x86-64).
2. It executes the special instruction syscall (x86-64) or svc (ARM).
3. The CPU traps to kernel mode, switching the stack to the kernel stack and jumping to the syscall handler.
4. The kernel validates arguments (pointers, permissions), performs the operation, and sets a result.
5. The handler returns to user mode with the syscall number’s return value in a register.

Each OS exposes a fixed ABI table of system services. On Linux x86-64, for instance, 0 = read, 1 = write, 2 = open, 57 = fork, 59 = execve, and so on; on Windows, the NT kernel dispatches through an analogous table of Nt*/Zw* system services. This ABI is a contract — it must never change, which is why adding a syscall is additive (new numbers) rather than a renumbering.

Syscall vs Library Call

A library call (read() in the C library, printf, malloc) may do all its work in user space or wrap one or more syscalls (the same layering exists in every OS’s runtime — glibc on Linux, the CRT on Windows):

This layering is why tracing library calls (ltrace) and tracing syscalls (strace) show different pictures, and why buffered I/O is so much faster than unbuffered: each syscall costs a mode switch, and buffering amortizes that cost across many logical operations.

The Cost of a System Call

A syscall is expensive relative to a function call because the CPU must:

• switch privilege levels (kernel/user) and re-validate state,
• switch stacks,
• flush or save parts of the pipelined CPU state,
• and (historically) flush the TLB.

A single syscall costs on the order of ~100 ns to ~1 µs — thousands of times a plain function call. At scale this dominates: a high-throughput server issuing one syscall per small I/O operation can be syscall-bound. This is the deep reason behind buffering, memory-mapped I/O, batched/asynchronous I/O (Linux io_uring, Windows IOCP), zero-copy paths like sendfile, and the general engineering rule: batch your system calls.

Standard Interfaces: POSIX and Win32

No application calls the raw syscall ABI directly — it programs against a standardized interface. POSIX normalizes the API across Unix-like systems (Linux, BSD, macOS): code written against open, read, fork, pthread_create is portable to any POSIX system because those names map to each OS’s syscall ABI. Windows exposes the analogous Win32/Win64 API (plus the lower-level NT Nt*/Zw* system services), which serves the same role on NT-family systems. The categories are the same across both — only the names differ:

The abstraction layer above these interfaces (the VFS for files, sockets for networking) is why the same read() works on a file, a socket, and a device — the kernel funnels them through one interface.

Error Handling Convention

System calls report failure through a return convention rather than exceptions: POSIX returns -1 and sets errno; Windows returns an NTSTATUS/HRESULT value (with GetLastError() for the Win32 view). Either way, every call must be checked. On POSIX, EINTR (interrupted by a signal) and EAGAIN (would block) are not fatal errors but conditions to retry, and mishandling them is a classic source of subtle bugs. This “every call returns an error” style is the ancestor of Go’s explicit error returns and Rust’s Result.

Tracing System Calls

Tracing tools intercept every syscall a process makes, printing arguments and results. On Linux/BSD, strace is the standard tool:

strace ls
strace -e trace=open,read ls            # filter
strace -p 1234                          # attach to a running process
strace -c ls                            # syscall summary (count + time)

The same class of tooling exists on other platforms — dtrace/dtruss on macOS, and Process Monitor / ETW tracing on Windows. strace -c is the fastest way to answer “why is this slow?”: it shows which syscalls dominate, exposing pathological patterns like a syscall per byte, repeated stat calls, or constant connect retries. It is the first tool for the Debugging & Profiling topic, and the mental model of “every program is a stream of syscalls” underpins everything from performance engineering to container security.

Practice Trajectory

1. Run strace -c ls and name the top three syscalls; explain why read/write/stat dominate. On macOS use dtruss, on Windows Process Monitor.
2. Compare the syscall count of a buffered vs unbuffered C program writing 10 MB (strace -c on Unix, equivalent I/O tracing on Windows) — watch the count collapse with buffering.
3. Trace a server you run and identify a single hot syscall to optimize (e.g., replacing read+write with sendfile or async I/O on Linux, IOCP on Windows).
4. Find how your OS’s ABI table is declared — on Linux, grep -E "syscall|nr=" /usr/include/asm/unistd_64.h (confirm read is entry 0); on Windows, inspect the NT Nt* service dispatch in ntdll or the system-service documentation.
5. Trace a program that handles an interruptible blocking call: interrupt a blocking read with a signal (kill -INT on Unix) or cancel it on Windows, and observe the retry loop.

When It’s the Right Tool