x86-64 System Call ABI

Coverage: syscall instruction → argument register conventions → red zone → vDSO → vsyscall → historical comparison with int 0x80 / sysenter Applicable to: x86-64 (AMD64), Linux 3.x+

Overview

x86-64 system calls use the syscall instruction (Intel syntax: syscall). This is the fast system call mechanism introduced by AMD64, replacing the three-generation evolution of 32-bit x86 (int 0x80 → sysenter/sysexit → syscall/sysret). Understanding the calling convention is useful for writing hand-crafted assembly, inline asm, or simply reading syscall instructions in disassembly.

Register Conventions

# System call number:
  rax = syscall number ( __NR_read = 0, __NR_write = 1, ...)

# Arguments (up to 6):
  rdi = arg1    rsi = arg2    rdx = arg3
  r10 = arg4    r8  = arg5    r9  = arg6

# Return value:
  rax = return value (>=0: success, -errno: error → user-space libc converts to errno)
  rdx = second return value (only for a few syscalls)

# Clobbered (modified by syscall):
  rcx = saved RIP (syscall instruction writes RCX ← RIP)
  r11 = saved RFLAGS (syscall instruction writes R11 ← RFLAGS)

# Preserved (not modified by syscall):
  rbx, r12-r15, rbp, rsp

Note that r10 replaces rcx as the 4th argument because the syscall instruction uses rcx to save the return address.

syscall Instruction Behavior

syscall:
  1. RCX ← RIP   (save user-space return address)
  2. R11 ← RFLAGS
  3. RIP ← IA32_LSTAR (MSR 0xC0000082, points to entry_SYSCALL_64)
  4. CS  ← IA32_STAR[47:32]
  5. SS  ← IA32_STAR[47:32] + 8
  6. Switch to Ring 0 (kernel mode)
  → Kernel now running with kernel stack, IF unchanged

sysretq (return):
  1. RIP ← RCX   (restore user-space next instruction)
  2. RFLAGS ← R11 (restore flags register, lower 32 bits)
  3. Switch to Ring 3 (user mode)

Key point: syscall does not automatically switch stacks—the kernel must manually switch to the kernel stack in entry_SYSCALL_64 via swapgs + reading the per-CPU kernel_stack.

Kernel Entry: entry_SYSCALL_64

# arch/x86/entry/entry_64.S
entry_SYSCALL_64:
    swapgs                    # Switch GS base from user-space to kernel-space per-CPU area
    movq  %rsp, PER_CPU(cpu_tss_rw + TSS_sp0)  # Save user stack to TSS
    movq  PER_CPU(pcpu_hot + X_top_of_stack), %rsp  # Switch to kernel stack

    pushq $__USER_DS          # Construct iret frame (SS)
    pushq PER_CPU(cpu_tss_rw + TSS_sp0)  # (RSP)
    pushq %r11                # (RFLAGS)
    pushq $__USER_CS          # (CS)
    pushq %rcx                # (RIP)

    # Call do_syscall_64(rdi=pt_regs, rax=nr)
    call do_syscall_64

Red Zone: A Trap in the x86-64 ABI

x86-64 ABI: 128 bytes below the stack pointer (RSP) is the "red zone"
   → Signal handlers must not touch these 128 bytes
   → Compilers can use this area without adjusting RSP (leaf functions)

Interaction with the kernel:
  Entering the kernel: The kernel places an iret frame below RSP → it does not touch the red zone
  Returning to user-space: User-space may rely on data in the red zone → kernel must not corrupt it

  Signal handling: When installing a sigframe, the red zone must be considered
    → Signal stack frame starts below RSP-128

vDSO: System Calls Without Entering the Kernel

// arch/x86/entry/vdso/
// Some system calls are executed directly in user-space via vDSO:

// __vdso_clock_gettime(): Read TSC + apply timekeeper correction → no syscall!
// __vdso_getcpu():        Read per-CPU variable → no syscall!
// __vdso_time():          Read timekeeper cache → no syscall!
// __vdso_getrandom():     Read kernel-maintained random pool → no syscall!

// vsyscall (old, deprecated):
//   Fixed at 0xffffffffff600000 → security risk (fixed address = ROP target)
//   Now emulated by default (via page fault emulation) → very slow → should use vDSO

Comparison with 32-bit x86

                         int 0x80        sysenter         syscall (x86-64)
Architecture              x86             x86 (Pentium II+)  x86-64 (AMD64)
Save return address       None (software) Hardware: ECX ← EIP   Hardware: RCX ← RIP
CPU context switch        Via IDT         Via MSR            Via MSR
Return value              eax             eax                rax
Clobbered registers       Few             ecx, r11           rcx, r11
Latency (cycles)          ~150             ~80                ~50-70

Complete System Call Example

# Hand-written write(1, "hello\n", 6) — x86-64 Linux
    movq $1, %rax        # __NR_write = 1
    movq $1, %rdi        # fd = stdout
    leaq msg(%rip), %rsi # buf
    movq $6, %rdx        # count
    syscall

    movq $60, %rax       # __NR_exit = 60
    xorq %rdi, %rdi      # exit code = 0
    syscall

msg: .ascii "hello\n"

References

• Source Code: arch/x86/entry/entry_64.S, arch/x86/entry/vdso/, arch/x86/include/asm/syscall.h
• ABI Documentation: System V AMD64 ABI, Linux x86-64 syscall table: /usr/include/asm/unistd_64.h
• LWN: "The vDSO and vsyscall", "Faster syscalls"

Keywords: syscall, sysret, entry_SYSCALL_64, red zone, vDSO, vsyscall, swapgs, calling convention