Writing an OS Kernel from Scratch | System Calls — How User-Mode Triggers Kernel Code
You wrote a single line write(fd, "hello", 5); in your program. This is just a C function call inside libc — how does it make its way step by step to the kernel, ultimately writing characters to the screen or a file?
This is the world of System Calls: the only entry point for user-mode programs to request kernel services. It’s not as simple as a regular function call — there’s a hardware-level wall (Ring switching) between user mode and kernel mode, and system calls are the only legitimate passage through that wall.
Today, let’s figure out how this wall is built and why write and open must go through system calls, while printf only works in user mode.
1. Why Can’t You Directly Jump to the Kernel
Regular function call: call func → jump to function address → execute → ret return
This path executes entirely in user mode. Ring 3 code can only jump to code that is also at Ring 3. What if you want to jump to Ring 0 kernel code?
A direct jump won’t work. Reasons:
-
CPU Protected Mode: Segment descriptors have a DPL (Descriptor Privilege Level) field. User-mode code’s CS/DS selectors have DPL=3, kernel code has DPL=0. If you try to use
jmporcallto jump to DPL=0 code, the CPU triggers #GP (General Protection Fault) -
Page-Level Privilege: Kernel pages are marked as Supervisor mode (U/S=0) in page tables. User-mode (U/S=1) code accessing these pages triggers #PF
-
Isolation Requirements: If any user-mode code could jump to the kernel, the entire system security model collapses — malicious programs could directly read and write arbitrary memory
So system calls must go through a channel specifically designed by the CPU for this purpose.
2. Three Implementation Methods of System Calls
x86 has historically seen three ways to implement system calls:
2.1 Software Interrupt (INT 0x80) — The Classic
System call method before Linux 2.6:
User mode:
mov eax, 4 ; System call number (write = 4)
mov ebx, fd ; Parameter 1
mov ecx, buf ; Parameter 2
mov edx, len ; Parameter 3
int 0x80 ; Trigger software interrupt, CPU automatically looks up IDT[0x80]
Kernel mode (IDT[0x80] points to sys_call_table):
handler_0x80:
push registers...
call [sys_call_table + eax*4] ; Use call number as index
pop registers...
iret ; Return to user modeINT 0x80 workflow:
- CPU checks IF flag (ensures interrupts are enabled)
- Reads gate descriptor from IDT[0x80]
- Verifies CPL <= DPL (Current Privilege Level <= Descriptor Privilege Level)
- If it’s an interrupt gate, automatically disables IF
- Saves CS/EIP/EFLAGS to stack (or switches stack for task gates)
- Jumps to kernel sys_call_table
Disadvantage: INT is a bidirectional gate (can enter and exit), but every system call requires looking up the IDT and verifying privileges, adding extra overhead
2.2 SYSENTER / SYSCALL — Fast System Calls
CPU vendors realized software interrupts were too slow and specifically designed fast call instructions:
SYSENTER (Intel) / SYSCALL (AMD):
User mode (after setting call parameters):
mov eax, 4 ; System call number
mov ebx, fd
mov ecx, buf
mov edx, len
sysenter ; Intel-specific, fast jump
Preset in kernel (MSR registers):
IA32_SYSENTER_EIP = address of sys_call_handler
IA32_SYSENTER_CS = Kernel code segment selector
IA32_SYSENTER_ESP = Kernel stack pointer
When SYSENTER executes:
→ Direct jump to IA32_SYSENTER_EIP (no IDT lookup needed)
→ CS = IA32_SYSENTER_CS
→ ESP = IA32_SYSENTER_ESP (new kernel stack)
→ Auto-switch to Ring 0SYSCALL (AMD):
When SYSCALL executes:
→ Saves RIP → RCX (for return)
→ Saves RFLAGS → R11
→ Jumps to IA32_LSTAR (MSR register, system call entry address)
→ Sets CS = IA32_STAR[47:32] (kernel CS)
→ Sets SS = IA32_STAR[47:32] + 8 (kernel SS)
→ Auto-switch to Ring 0
Return instruction: SYSRET
→ RCX → RIP (restore return address)
→ R11 → RFLAGS (restore flags)
→ Restore user-mode CS/SSModern Linux uses SYSCALL (x86_64). Intel later also added SYSCALL support, so modern x86_64 uses the unified SYSCALL/SYSRET.
2.3 Performance Comparison
System call performance benchmark:
Method Cycles (approx) Notes
INT 0x80 ~300-400 Must traverse IDT, check permissions, save many values
SYSENTER ~100-200 Direct MSR jump, reduced save/restore
SYSCALL ~70-100 Optimized specifically for 64-bit mode3. Linux System Call Table
Linux organizes all system calls into a table (sys_call_table), indexed by call number:
Partial sys_call_table (x86_64):
0 sys_read — Read file descriptor
1 sys_write — Write file descriptor
2 sys_open — Open file
3 sys_close — Close file
9 sys_mmap — Memory mapping
12 sys_brk — Change data segment size
39 sys_getpid — Get process ID
56 sys_clone — Create process
57 sys_fork — Create child process
59 sys_execve — Execute program
60 sys_exit — Exit process
61 sys_wait4 — Wait for process4. The Complete Path of a System Call
Take write(1, "Hello", 5) as an example, tracing the full path:
User Program
↓
libc wrapper: write(1, "Hello", 5)
↓ (libc handles the details, fills registers)
User-mode system call entry:
mov eax, 1 ; SYS_write = 1 (x86_64)
mov edi, 1 ; fd = 1 (stdout)
mov rsi, buf ; buf points to "Hello"
mov rdx, 5 ; count = 5
syscall ; ⟶ enters kernel
↓
Kernel entry: entry_SYSCALL_64
├── Save all registers to kernel stack (pt_regs)
├── Verify system call number is valid
├── Call sys_write from sys_call_table[1]
│ ├── fsnotify (file system notification)
│ ├── Get file struct by fd
│ ├── Call file->f_op->write (e.g., tty_write for stdout)
│ │ ├── tty_write → n_tty_write → copy from user space
│ │ ├── Write data to terminal buffer
│ │ └── Schedule display output
│ └── Return number of bytes written
├── Check if rescheduling is needed
└── SYSRET return to user mode
↓
Return to libc, back to user program5. Parameter Passing and Validation
5.1 Parameter Conventions
x86_64 Linux system call convention (different from regular function calls!):
Parameter syscall function call
Arg 1: rdi rdi
Arg 2: rsi rsi
Arg 3: rdx rdx
Arg 4: r10 rcx (rcx is used by syscall to save RIP)
Arg 5: r8 r8
Arg 6: r9 r9
Return: rax rax5.2 User Pointer Validation
The kernel must never blindly trust pointers passed from user space:
// Kernel check for user-space pointers (simplified)
// Source: include/linux/uaccess.h
#define access_ok(addr, size)
((unsigned long)(addr) < TASK_SIZE_MAX)
// Safely copy data from user space
unsigned long copy_from_user(void *to, const void __user *from, unsigned long n) {
if (!access_ok(from, n))
return n; // Return un-copied count (error)
// Use special instructions (e.g., movs with exception handling)
// Automatically handles page faults and invalid addresses
return __copy_from_user(to, from, n);
}copy_from_user is special because:
- It checks if the pointer is in user-space range
- It uses CPU instructions with fault handling capabilities
- If a page fault occurs (e.g., the address page hasn’t been loaded), the kernel automatically handles it
- It returns how many bytes were NOT copied, so you can tell if it succeeded
6. Summary
- System calls are the only way for user mode to access kernel services
- Three methods have existed historically: INT 0x80 (old), SYSENTER (intermediate), and SYSCALL (modern standard)
- Linux uses SYSCALL on x86_64, stored as a function pointer table sys_call_table
- glibc acts as a wrapper hiding hardware details; you just call write(), fopen() and it works
- User pointers must be validated — copy_from_user ensures security
- System call = synchronous: the calling process waits for the kernel to finish before returning
Next up: How processes are born — fork and exec in detail.
Comments