Linux x86-64 System Call ABI

When writing hand-crafted assembly, inline asm, debugging seccomp, or reading disassembly, the most confusing aspect is the existence of three different sets of rules:

1. System V AMD64 C ABI: How normal user-space functions pass arguments, return values, and save registers.
2. Linux x86-64 syscall ABI: How user-space passes the system call number and up to 6 arguments to the kernel.
3. CPU SYSCALL/SYSRET instruction semantics: Which registers the hardware actually modifies during privilege level transitions.

This article explains these three layers separately, discusses the native LP64 ABI, and does not cover the x32 ABI. The implementation example uses the x86-64 entry code from Linux v6.18. Kernel implementations evolve; when copying source code snippets, you should note the version number rather than treating details from a specific version as a permanent ABI.

1. Linux Syscall Register Conventions

Typical invocation:

mov $1, %rax          # __NR_write
mov $1, %rdi          # fd = STDOUT_FILENO
lea message(%rip), %rsi
mov $6, %rdx
syscall

For write here, a successful call returns the number of bytes written, while a failure returns -errno. The native Linux ABI uses -4095 to -1 to indicate errors; you cannot uniformly interpret all successful system call results as byte counts. glibc wrapper functions (including the public syscall()) convert kernel errors to -1 and set the thread-local errno.

Why is the 4th argument r10 instead of rcx?

For normal System V functions, the 4th integer argument uses rcx; however, the SYSCALL instruction must write the user-space return address into rcx. Therefore, the Linux syscall ABI moves the 4th argument to r10.

This is also why inline asm must declare rcx and r11 as clobbered:

static inline long raw_write(int fd, const void *buf, unsigned long count)
{
    register long rax __asm__("rax") = 1;   /* __NR_write */
    register long rdi __asm__("rdi") = fd;
    register long rsi __asm__("rsi") = (long)buf;
    register long rdx __asm__("rdx") = count;

    __asm__ volatile (
        "syscall"
        : "+a"(rax)
        : "D"(rdi), "S"(rsi), "d"(rdx)
        : "rcx", "r11", "memory"
    );
    return rax;
}

For arguments 4–6, you need to explicitly bind r10, r8, and r9. Production code should generally prefer calling libc; raw syscalls bypass cancellation points, compatibility wrappers, and libc error handling.

Complete six-argument example: Allocate a page of anonymous memory and then free it

Save as raw-mmap.c and compile with GCC or Clang on native Linux x86-64. This example retains libc for printing and querying page size, but mmap/munmap are executed via inline syscalls. The SYS_* macros come from the native target headers to avoid accidentally moving a call number to a different ABI.

#include <errno.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <unistd.h>

static long raw6(long number, long a1, long a2, long a3,
                 long a4, long a5, long a6) {
    register long r10 __asm__("r10") = a4;
    register long r8 __asm__("r8") = a5;
    register long r9 __asm__("r9") = a6;
    long result;
    __asm__ volatile("syscall"
        : "=a"(result)
        : "0"(number), "D"(a1), "S"(a2), "d"(a3),
          "r"(r10), "r"(r8), "r"(r9)
        : "rcx", "r11", "memory");
    return result;
}
static int raw_error(long result) {
    return (unsigned long)result >= (unsigned long)-4095L;
}
int main(void) {
    long size = sysconf(_SC_PAGESIZE);
    if (size <= 0) return 1;
    long address = raw6(SYS_mmap, 0, size, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (raw_error(address)) {
        fprintf(stderr, "mmap error=%ld\n", -address);
        return 1;
    }
    unsigned char *page = (unsigned char *)address;
    page[0] = 7;
    page[size - 1] = 9;
    int valid = page[0] == 7 && page[size - 1] == 9;
    long released = raw6(SYS_munmap, address, size, 0, 0, 0, 0);
    if (raw_error(released) || !valid) return 1;

    errno = 0;
    long bad = raw6(SYS_mmap, 0, 0, PROT_READ,
                   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    int unchanged = errno == 0;
    if (bad != -EINVAL || !unchanged) return 1;
    printf("page checked; raw error=%ld; errno unchanged=%d\n",
           -bad, unchanged);
    return 0;
}

cc -std=gnu11 -Wall -Wextra -O2 raw-mmap.c -o raw-mmap
./raw-mmap
objdump -d -Mintel raw-mmap

For the first call, the flags are the 4th argument and must go into r10; fd=-1 and offset=0 go into r8 and r9 respectively. Anonymous mappings return zero-initialized memory; this example writes the first and last bytes and then releases it. The second mmap with a length of zero should return -EINVAL and not set errno on behalf of the caller. mmap(2)

Error detection first casts the result to unsigned long and compares it with the unsigned representation of -4095, interpreting the top reserved range as an error. Do not dereference the address result as a valid pointer before checking, and do not assume errno will be automatically filled by the kernel; that is the job of the libc wrapper layer. raw6 is merely a demonstration of synchronous return calls in this example and should not be used without modification for interfaces like clone or rt_sigreturn that change the stack/control flow. syscall(2)

Local register variables combined with extended asm constraints place inputs in specific registers; the declaration itself does not represent permanent exclusive register ownership. Therefore, complete argument calculations first, then immediately execute the asm; do not insert function calls in between that might overwrite caller-saved registers. GCC Local Register Variables

Return values must be interpreted according to the specific call

For example, if you want to write 6 bytes and the returns are sequentially 2, -EINTR, 4, you should not advance the pointer by 2 again on the second call: only positive return values confirm progress in that specific attempt. The assembly loop at the end of this article retains rsi/rdx and retries the current remainder upon -EINTR, updating these two registers only for short writes.

The two identities of rcx cannot be preserved simultaneously

There is no need to memorize two separate register tables: letting the 4th argument move along the rcx→r10 path and then observing the hardware's overwrite of rcx reveals why the two ABIs must connect. RFLAGS* represents the flag values after instruction-defined processing; see the next section for details on RF.

2. What Does the SYSCALL Instruction Do?

When executing SYSCALL in 64-bit mode, the CPU primarily performs:

Two key conclusions:

• rcx and r11 are occupied by hardware and cannot be used to save values across syscalls.
• SYSCALL does not save user-space rsp, nor does it automatically switch to the kernel stack.

The diagram above omits special handling for RF: the hardware first clears RF, then saves the corresponding flag status to R11. RFLAGS is not simply "kept unchanged." The kernel configures the flags to be cleared in IA32_FMASK; the Linux entry code manages interrupts after establishing a safe state.

3. Linux entry_SYSCALL_64

The core sequence of the x86-64 entry in Linux v6.18 is:

# arch/x86/entry/entry_64.S (security hardening and annotations omitted)
entry_SYSCALL_64:
    swapgs
    movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)  # Save user rsp temporarily
    SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
    movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp

    pushq $__USER_DS
    pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
    pushq %r11
    pushq $__USER_CS
    pushq %rcx
    pushq %rax
    # Continue constructing struct pt_regs, then call do_syscall_64

Here, tss.sp2 is a per-CPU scratch slot used by Linux, not the traditional sp0 flow where the hardware automatically switches stacks. The order is also important: save user rsp first, then switch kernel page tables and kernel stack, and finally construct struct pt_regs on the kernel stack.

Therefore, the user-space red zone is not overwritten by the register frame on a normal syscall entry; the frame is located on the kernel stack, not "below the user rsp."

4. Return: SYSRET Fast Path and IRET Fallback

After a system call, Linux checks whether the return context is suitable for SYSRET. The fast path is taken only if conditions such as address, segment registers, and flag bits are all safe; otherwise, IRET is used.

This is not merely a performance choice. AMD and Intel's SYSRET have difficult-to-handle behaviors for edge cases such as non-canonical addresses; the kernel must first verify user-controllable pt_regs.

From the user-space perspective, both return paths should satisfy the same syscall ABI; you cannot rely on the kernel definitely using SYSRET.

5. Red Zone and System Calls

The System V AMD64 ABI defines a 128-byte red zone below the user rsp. Leaf functions can temporarily use this space without adjusting rsp.

Three situations need to be distinguished:

• Normal syscall/interrupt entry into kernel: Linux switches to the kernel stack; the entry frame does not write to the user red zone.
• Delivering a signal to user space: The kernel allocates space for the red zone when constructing the signal frame.
• Kernel code itself: The kernel is compiled with -mno-red-zone and cannot rely on the user-space ABI's red zone.

When writing hand-crafted inline asm, the "memory" clobber only constrains compiler memory reordering; it does not imply that the syscall only accesses a fixed memory range. Which buffers the pointers passed to the kernel point to are still defined by the specific syscall.

6. vDSO is Not "Executing Syscall in User Space"

The vDSO is a small piece of ELF code mapped by the kernel into the process address space. Some libc APIs can first call the vDSO to calculate results from read-only data maintained by the kernel, thereby avoiding the actual execution of SYSCALL.

Common symbols include:

• __vdso_clock_gettime
• __vdso_gettimeofday
• __vdso_time
• __vdso_getcpu
• __vdso_getrandom on newer kernels

The existence of these symbols and their ability to fulfill requests depend on the architecture and kernel version; libc must be prepared to fall back to a real syscall.

Check the vDSO for the current process:

rg '\[vdso\]' /proc/self/maps

# In a glibc environment, you can observe whether the actual call still enters the kernel
strace -e trace=clock_gettime,gettimeofday /bin/date

7. int 0x80, sysenter, and 64-bit syscall

The three belong to different entry mechanisms and cannot be compared using a fixed "cycle count" table:

Actual latency is affected by CPU microarchitecture, KPTI, mitigation measures, virtualization, frequency, and kernel version. When performance data is needed, it should be measured on the target machine, and the environment should be reported:

perf stat -r 20 ./syscall-benchmark

Do not treat 50–70 cycles on an old CPU as a cross-platform fact.

8. Complete Assembly Example

.section .rodata
message:
    .ascii "hello\n"

.section .text
.global _start
_start:
    lea message(%rip), %rsi
    mov $6, %rdx
.write:
    mov $1, %rax          # __NR_write
    mov $1, %rdi          # stdout
    syscall
    cmp $-4, %rax        # -EINTR: nothing written this time, retry
    je .write
    test %rax, %rax
    jle .error           # Negative value or no progress; this example does not wait for EAGAIN
    add %rax, %rsi
    sub %rax, %rdx
    jnz .write           # Short write: continue writing the remainder
    xor %rdi, %rdi
    jmp .exit
.error:
    mov $1, %rdi
.exit:
    mov $60, %rax        # __NR_exit: Exit current thread
    syscall
.section .note.GNU-stack,"",@progbits

The example is a single-threaded program without libc; the original exit exits the current thread, and exit_group exits the entire thread group. If stdout is a pipe with the read end closed, the default SIGPIPE may also terminate the process directly.

Build and observe:

as -o syscall.o syscall.S
ld -o syscall syscall.o
strace ./syscall
objdump -drwC -Mintel ./syscall

9. Authoritative Resources

• Linux v6.18 entry_SYSCALL_64
• Linux x86-64 System Call Number Table
• Linux man-pages syscall(2)
• x86-64 psABI
• Intel 64 and IA-32 Architectures Software Developer Manuals

Recommended reading order: First confirm the Linux ABI using syscall(2), then confirm the SYSCALL/SYSRET instruction semantics using the processor manual, and finally cross-reference entry_64.S for the target kernel version. This prevents confusing CPU rules, user-space C ABI, and a specific kernel implementation.