Linux x86-64 System Call ABI

When writing hand-crafted assembly, inline asm, debugging seccomp, or reading disassembly, the most confusing aspect is the existence of three different sets of rules:

1. System V AMD64 C ABI: How user-space functions pass arguments, return values, and save registers.
2. Linux x86-64 syscall ABI: How user-space passes the system call number and up to 6 arguments to the kernel.
3. CPU SYSCALL/SYSRET instruction semantics: Which registers the hardware actually modifies during privilege level transitions.

This article separates these three layers and uses the current Linux x86-64 entry code as a baseline. Kernel implementations evolve; when copying source code snippets, note the version rather than treating details from a specific version as a permanent ABI.

1. Linux syscall Register Conventions

Typical invocation:

mov $1, %rax          # __NR_write
mov $1, %rdi          # fd = STDOUT_FILENO
lea message(%rip), %rsi
mov $6, %rdx
syscall

The Linux kernel returns non-negative results directly or -errno on error. libc wrappers like glibc then convert -errno to -1 and set the thread-local errno.

Why is the 4th argument r10 instead of rcx?

For regular System V functions, the 4th integer argument uses rcx; however, the SYSCALL instruction must write the user-space return address into rcx. Therefore, the Linux syscall ABI moves the 4th argument to r10.

This is also why inline asm must declare rcx and r11 as clobbered:

static inline long raw_write(int fd, const void *buf, unsigned long count)
{
    register long rax __asm__("rax") = 1;   /* __NR_write */
    register long rdi __asm__("rdi") = fd;
    register long rsi __asm__("rsi") = (long)buf;
    register long rdx __asm__("rdx") = count;

    __asm__ volatile (
        "syscall"
        : "+a"(rax)
        : "D"(rdi), "S"(rsi), "d"(rdx)
        : "rcx", "r11", "memory"
    );
    return rax;
}

For arguments 4–6, you must explicitly bind r10, r8, and r9. Production code should generally prefer calling libc; raw syscalls bypass cancellation points, compatibility wrappers, and libc error handling.

2. What the SYSCALL Instruction Does

When executing SYSCALL in 64-bit mode, the CPU primarily performs:

RCX    <- Next user-space instruction address
R11    <- User-space RFLAGS
RIP    <- IA32_LSTAR
RFLAGS <- RFLAGS & ~IA32_FMASK
CS/SS  <- Kernel selectors derived from IA32_STAR
CPL    <- 0

Two key conclusions:

• rcx and r11 are occupied by hardware and cannot be used to save values across syscalls.
• SYSCALL does not save the user-space rsp nor automatically switch to the kernel stack.

RFLAGS is not simply "kept unchanged." The kernel configures flags to be cleared in IA32_FMASK; the Linux entry code manages interrupts after establishing a safe state.

3. Linux entry_SYSCALL_64

The core sequence of the current x86-64 entry point is:

# arch/x86/entry/entry_64.S (security hardening and annotations omitted)
entry_SYSCALL_64:
    swapgs
    movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)  # Save user rsp temporarily
    SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
    movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp

    pushq $__USER_DS
    pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
    pushq %r11
    pushq $__USER_CS
    pushq %rcx
    pushq %rax
    # Continue constructing struct pt_regs, then call do_syscall_64

Here, tss.sp2 is a per-CPU scratch slot used by Linux, not the traditional sp0 flow where hardware automatically switches stacks. The order is also important: first save the user rsp, then switch to the kernel page table and kernel stack, and finally construct struct pt_regs on the kernel stack.

Therefore, the user-space red zone is not overwritten by the register frame at the normal syscall entry; the frame is located on the kernel stack, not "below the user rsp."

4. Return: SYSRET Fast Path and IRET Fallback

After a system call, Linux checks if the return context is suitable for SYSRET. The fast path is taken only if conditions such as address, segment registers, and flags are safe; otherwise, IRET is used.

This is not merely a performance choice. AMD and Intel's SYSRET have problematic behaviors for edge cases like non-canonical addresses, so the kernel must first validate user-controlled pt_regs.

From the user-space perspective, both return paths must satisfy the same syscall ABI; you cannot rely on the kernel always using SYSRET.

5. Red Zone and System Calls

The System V AMD64 ABI defines a 128-byte red zone below the user rsp. Leaf functions can temporarily use this space without adjusting rsp.

Three situations need to be distinguished:

• Normal syscall/interrupt entry into the kernel: Linux switches to the kernel stack; the entry frame does not write to the user red zone.
• Signal delivery to user space: The kernel allocates space for the red zone when constructing the signal frame.
• Kernel code itself: The kernel is compiled with -mno-red-zone and cannot rely on the user-space ABI's red zone.

When writing hand-crafted inline asm, the "memory" clobber only constrains compiler memory reordering; it does not imply that the syscall only accesses a fixed memory range. Which buffers the pointers passed to the kernel point to are still defined by the specific syscall.

6. vDSO is Not "Executing Syscalls in User Space"

The vDSO is a small piece of ELF code mapped into the process address space by the kernel. Some libc APIs can first call the vDSO to calculate results from read-only data maintained by the kernel, thereby avoiding the actual execution of SYSCALL.

Common symbols include:

• __vdso_clock_gettime
• __vdso_gettimeofday
• __vdso_time
• __vdso_getcpu
• __vdso_getrandom on newer kernels

Whether these symbols exist and can fulfill the request depends on the architecture and kernel version; libc must be prepared to fall back to the actual syscall.

Check the vDSO for the current process:

grep '\[vdso\]' /proc/self/maps

# In a glibc environment, you can observe whether actual calls still enter the kernel
strace -e trace=clock_gettime,gettimeofday /bin/date

7. int 0x80, sysenter, and 64-bit syscall

These three belong to different entry mechanisms and cannot be compared using a fixed "cycle count" table:

Actual latency is affected by CPU microarchitecture, KPTI, mitigations, virtualization, frequency, and kernel version. When performance data is needed, measure on the target machine and report the environment:

perf stat -r 20 ./syscall-benchmark

Do not treat 50–70 cycles on an old CPU as a cross-platform fact.

8. Complete Assembly Example

.section .rodata
message:
    .ascii "hello\n"

.section .text
.global _start
_start:
    mov $1, %rax          # __NR_write
    mov $1, %rdi          # stdout
    lea message(%rip), %rsi
    mov $6, %rdx
    syscall

    test %rax, %rax
    js .error

    xor %rdi, %rdi
    jmp .exit

.error:
    mov $1, %rdi
.exit:
    mov $60, %rax         # __NR_exit
    syscall

Build and observe:

as -o syscall.o syscall.S
ld -o syscall syscall.o
strace ./syscall
objdump -drwC -Mintel ./syscall

9. Authoritative Resources

• Linux entry_SYSCALL_64 current source code
• Linux x86 syscall argument mapping
• Linux man-pages syscall(2)
• x86-64 psABI
• Intel 64 and IA-32 Architectures Software Developer Manuals

Recommended reading order: First use syscall(2) to confirm the Linux ABI, then use the processor manual to confirm the SYSCALL/SYSRET instruction semantics, and finally compare with entry_64.S for the target kernel version. This prevents confusing CPU rules, user-space C ABI, and a specific kernel implementation.