---
title: Linux x86-64 System Call ABI
url: https://doc.liz6.com/en/systems-programming/02-system-call-abi/01-x86-64-system-call-abi
locale: en
area: systems-programming
tags:
- systems-programming
- system-call-abi
date: null
modified: 2026-08-11
description: Linux x86-64 System Call ABI When writing hand-crafted assembly, inline asm, debugging seccomp, or reading disassembly, the most confusing aspect is the existen…
---

# Linux x86-64 System Call ABI

When writing hand-crafted assembly, inline asm, debugging seccomp, or reading disassembly, the most confusing aspect is the existence of three different sets of rules:

1. **System V AMD64 C ABI**: How user-space functions pass arguments, return values, and save registers.
2. **Linux x86-64 syscall ABI**: How user-space passes the system call number and up to 6 arguments to the kernel.
3. **CPU `SYSCALL/SYSRET` instruction semantics**: Which registers the hardware actually modifies during privilege level transitions.

This article separates these three layers and uses the current Linux x86-64 entry code as a baseline. Kernel implementations evolve; when copying source code snippets, note the version rather than treating details from a specific version as a permanent ABI.

## 1. Linux syscall Register Conventions

| Purpose | Register |
|---|---|
| System call number | `rax` |
| Arguments 1–3 | `rdi`, `rsi`, `rdx` |
| Arguments 4–6 | `r10`, `r8`, `r9` |
| Return value | `rax` |
| Hardware clobbered | `rcx`, `r11` |

Typical invocation:

```asm
mov $1, %rax          # __NR_write
mov $1, %rdi          # fd = STDOUT_FILENO
lea message(%rip), %rsi
mov $6, %rdx
syscall
```

The Linux kernel returns non-negative results directly or `-errno` on error. libc wrappers like glibc then convert `-errno` to `-1` and set the thread-local `errno`.

### Why is the 4th argument `r10` instead of `rcx`?

For regular System V functions, the 4th integer argument uses `rcx`; however, the `SYSCALL` instruction must write the user-space return address into `rcx`. Therefore, the Linux syscall ABI moves the 4th argument to `r10`.

This is also why inline asm must declare `rcx` and `r11` as clobbered:

```c
static inline long raw_write(int fd, const void *buf, unsigned long count)
{
    register long rax __asm__("rax") = 1;   /* __NR_write */
    register long rdi __asm__("rdi") = fd;
    register long rsi __asm__("rsi") = (long)buf;
    register long rdx __asm__("rdx") = count;

    __asm__ volatile (
        "syscall"
        : "+a"(rax)
        : "D"(rdi), "S"(rsi), "d"(rdx)
        : "rcx", "r11", "memory"
    );
    return rax;
}
```

For arguments 4–6, you must explicitly bind `r10`, `r8`, and `r9`. Production code should generally prefer calling libc; raw syscalls bypass cancellation points, compatibility wrappers, and libc error handling.

## 2. What the `SYSCALL` Instruction Does

When executing `SYSCALL` in 64-bit mode, the CPU primarily performs:

```text
RCX    <- Next user-space instruction address
R11    <- User-space RFLAGS
RIP    <- IA32_LSTAR
RFLAGS <- RFLAGS & ~IA32_FMASK
CS/SS  <- Kernel selectors derived from IA32_STAR
CPL    <- 0
```

Two key conclusions:

- `rcx` and `r11` are occupied by hardware and cannot be used to save values across syscalls.
- `SYSCALL` **does not save the user-space `rsp` nor automatically switch to the kernel stack**.

RFLAGS is not simply "kept unchanged." The kernel configures flags to be cleared in `IA32_FMASK`; the Linux entry code manages interrupts after establishing a safe state.

## 3. Linux `entry_SYSCALL_64`

The core sequence of the current x86-64 entry point is:

```asm
# arch/x86/entry/entry_64.S (security hardening and annotations omitted)
entry_SYSCALL_64:
    swapgs
    movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)  # Save user rsp temporarily
    SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
    movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp

    pushq $__USER_DS
    pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
    pushq %r11
    pushq $__USER_CS
    pushq %rcx
    pushq %rax
    # Continue constructing struct pt_regs, then call do_syscall_64
```

Here, `tss.sp2` is a per-CPU scratch slot used by Linux, not the traditional `sp0` flow where hardware automatically switches stacks. The order is also important: first save the user `rsp`, then switch to the kernel page table and kernel stack, and finally construct `struct pt_regs` on the kernel stack.

Therefore, the user-space red zone is not overwritten by the register frame at the normal syscall entry; the frame is located on the kernel stack, not "below the user `rsp`."

## 4. Return: `SYSRET` Fast Path and `IRET` Fallback

After a system call, Linux checks if the return context is suitable for `SYSRET`. The fast path is taken only if conditions such as address, segment registers, and flags are safe; otherwise, `IRET` is used.

This is not merely a performance choice. AMD and Intel's `SYSRET` have problematic behaviors for edge cases like non-canonical addresses, so the kernel must first validate user-controlled `pt_regs`.

From the user-space perspective, both return paths must satisfy the same syscall ABI; you cannot rely on the kernel always using `SYSRET`.

## 5. Red Zone and System Calls

The System V AMD64 ABI defines a 128-byte red zone below the user `rsp`. Leaf functions can temporarily use this space without adjusting `rsp`.

Three situations need to be distinguished:

- **Normal syscall/interrupt entry into the kernel**: Linux switches to the kernel stack; the entry frame does not write to the user red zone.
- **Signal delivery to user space**: The kernel allocates space for the red zone when constructing the signal frame.
- **Kernel code itself**: The kernel is compiled with `-mno-red-zone` and cannot rely on the user-space ABI's red zone.

When writing hand-crafted inline asm, the `"memory"` clobber only constrains compiler memory reordering; it does not imply that the syscall only accesses a fixed memory range. Which buffers the pointers passed to the kernel point to are still defined by the specific syscall.

## 6. vDSO is Not "Executing Syscalls in User Space"

The vDSO is a small piece of ELF code mapped into the process address space by the kernel. Some libc APIs can first call the vDSO to calculate results from read-only data maintained by the kernel, thereby avoiding the actual execution of `SYSCALL`.

Common symbols include:

- `__vdso_clock_gettime`
- `__vdso_gettimeofday`
- `__vdso_time`
- `__vdso_getcpu`
- `__vdso_getrandom` on newer kernels

Whether these symbols exist and can fulfill the request depends on the architecture and kernel version; libc must be prepared to fall back to the actual syscall.

Check the vDSO for the current process:

```bash
grep '\[vdso\]' /proc/self/maps

# In a glibc environment, you can observe whether actual calls still enter the kernel
strace -e trace=clock_gettime,gettimeofday /bin/date
```

## 7. `int 0x80`, `sysenter`, and 64-bit `syscall`

These three belong to different entry mechanisms and cannot be compared using a fixed "cycle count" table:

| Mechanism | Primary Use Case | Entry Configuration |
|---|---|---|
| `int 0x80` | Traditional i386 ABI; compatibility entry in x86-64 | IDT |
| `sysenter/sysexit` | 32-bit fast system calls | SYSENTER MSR |
| `syscall/sysret` | Native x86-64 Linux syscall ABI | STAR/LSTAR/FMASK MSRs |

Actual latency is affected by CPU microarchitecture, KPTI, mitigations, virtualization, frequency, and kernel version. When performance data is needed, measure on the target machine and report the environment:

```bash
perf stat -r 20 ./syscall-benchmark
```

Do not treat `50–70 cycles` on an old CPU as a cross-platform fact.

## 8. Complete Assembly Example

```asm
.section .rodata
message:
    .ascii "hello\n"

.section .text
.global _start
_start:
    mov $1, %rax          # __NR_write
    mov $1, %rdi          # stdout
    lea message(%rip), %rsi
    mov $6, %rdx
    syscall

    test %rax, %rax
    js .error

    xor %rdi, %rdi
    jmp .exit

.error:
    mov $1, %rdi
.exit:
    mov $60, %rax         # __NR_exit
    syscall
```

Build and observe:

```bash
as -o syscall.o syscall.S
ld -o syscall syscall.o
strace ./syscall
objdump -drwC -Mintel ./syscall
```

## 9. Authoritative Resources

- [Linux `entry_SYSCALL_64` current source code](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/entry/entry_64.S)
- [Linux x86 syscall argument mapping](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/entry/syscalls/syscall_64.tbl)
- [Linux man-pages `syscall(2)`](https://man7.org/linux/man-pages/man2/syscall.2.html)
- [x86-64 psABI](https://gitlab.com/x86-psABIs/x86-64-ABI)
- [Intel 64 and IA-32 Architectures Software Developer Manuals](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)

Recommended reading order: First use `syscall(2)` to confirm the Linux ABI, then use the processor manual to confirm the `SYSCALL/SYSRET` instruction semantics, and finally compare with `entry_64.S` for the target kernel version. This prevents confusing CPU rules, user-space C ABI, and a specific kernel implementation.