356 | System Calls: How User Mode Triggers Kernel Code

Key Insight: User programs can never directly read or write hardware. Even just printing a character requires asking the kernel to do it on their behalf. System calls are the only gateway for this “proxy” service.

1. Problem: What If a User-Mode Program Wants to Print a Character?

When a user types echo hello in a shell, what happens behind the scenes:

Shell (User Mode)
  → write() function (glibc library function)
  → Kernel sys_write (System Call)
  → Terminal Driver (Kernel)
  → Screen

write is a System Call.

The user-mode program thinks it’s calling an ordinary function, but in reality:

• The CPU transitions from user mode (CPL=3) to kernel mode (CPL=0)
• Executes kernel code, accessing protected hardware
• On return, restores the user-mode context and continues execution

System call = The only controlled channel between user mode and kernel mode.

2. The Evolution of System Calls on x86

The method of triggering system calls on the x86 architecture has gone through three generations:

2.1 Real Mode Era: INT 0x80

The earliest method was software interrupts:

// User-mode code (32-bit protected mode)
mov eax, 4          // sys_write system call number
mov ebx, 1         // fd = stdout
mov ecx, msg       // buffer address
mov edx, len       // length
int 0x80           // Trigger interrupt, CPU automatically jumps to IDT[0x80]

The CPU looks up gate 0x80 in the IDT (Interrupt Descriptor Table), obtains the kernel function address, and jumps there.

Drawback: Slow — each call requires looking up the IDT, pushing to the stack, and saving context.

2.2 Early Protected Mode: SYSENTER / SYSCALL

# 32-bit: Uses SYSENTER (Intel)
sysexit:
    mov ecx, edx        // Return address
    mov edx, esp        // User stack
    sysenter            // Jump into kernel (no interrupt table lookup)
    ; Returns to ecx

# 64-bit: Uses SYSCALL (AMD)
syscall
    ; Invented by AMD, later supported by Intel
    ; Faster: no IDT lookup, directly reads MSR

SYSENTER/SYSCALL is 3-5x faster than INT 0x80 because:

• No interrupt descriptor table lookup
• Reads the kernel entry address directly from an MSR (Model Specific Register)
• The jump is deterministic, with no lookup overhead

2.3 Modern Linux: All Use SYSCALL

Linux x86-64 uniformly uses the syscall instruction, with the entry point at MSR IA32_LSTAR (address 0xC0000082):

// Set during kernel initialization
wrmsr(0xC0000082, (unsigned long)syscall_entry);
// From then on, the syscall instruction jumps directly here

User-mode code:

// Directly use syscall
mov rax, 1          // sys_exit system call number
mov rdi, 0          // exit code
syscall             // Enter kernel syscall_entry

3. System Call Numbers: How Does It Know Which Function to Jump To?

Linux defines a system call number table:

0:   sys_read
1:   sys_write
2:   sys_open
3:   sys_close
4:   sys_newfstat
...
60:  sys_exit
231: sys_socketcall   // nested call

Inside user-mode glibc’s write(fd, buf, len):

// Simplified glibc source
ssize_t write(int fd, const void *buf, size_t count) {
    ssize_t ret;
    __asm__ volatile(
        "movq $1, %%rax "    // sysca... [truncated]