Target Code and ABI

A function call is not just a few instructions—register saving responsibilities, argument passing mechanisms, and stack frame layout are all hardcoded in the ABI; violating it causes immediate crashes. The differences between x86-64 System V and Win64 are the most common pitfalls in cross-platform compilation.

Overview

Instruction Selection maps IR operations to target machine instructions. However, a function call is not a simple mapping of a few instructions—it involves who is responsible for saving which registers, how arguments are passed, how the stack frame is laid out, and where the return address is stored. These conventions are all defined in an ABI (Application Binary Interface) specification, which both the operating system and compiler must adhere to. Violating the ABI causes the program to crash directly—not due to logical errors, but because the stack is corrupted and the return address is overwritten. This article focuses on x86-64 System V (Unix/Linux/Mac), comparing it with Win64 to clarify every part of the stack frame.

Calling Convention: Register and Stack Protocols

x86-64 System V argument passing:

Argument order: rdi, rsi, rdx, rcx, r8, r9, [Stack (right-to-left push)]
Return value:   rax (within 64-bit), rdx:rax (128-bit struct)
Caller-saved:   rax, rcx, rdx, rsi, rdi, r8-r11   ← may be destroyed after call
Callee-saved:   rbx, rbp, r12-r15                  ← callee must save and restore if used

A call to fn(a, b, c, d, e, f, g) under x86-64 System V:

mov rdi, a          ; Argument 1
mov rsi, b          ; Argument 2
mov rdx, c          ; Argument 3
mov rcx, d          ; Argument 4
mov r8, e           ; Argument 5
mov r9, f           ; Argument 6
push g              ; Argument 7 → Stack (pushed before call)
call fn

Win64 is different: the first 4 arguments use rcx, rdx, r8, r9, and the rest go on the stack. Additionally, the caller must reserve "shadow space" (32 bytes) on the stack for the first 4 arguments—this space exists even if not used. This is a common pitfall in cross-platform compilation.

Anatomy of the Stack Frame

Starting from rbp (frame pointer), positive offsets lead to the return address and caller's arguments; negative offsets lead to the callee's local variables. The frame pointer is optional—modern compilers often use frameless functions: accessing locals via rsp offsets only, without pushing rbp (saving instructions, but making stack frame backtracking difficult during debugging). Debug information (DWARF .debug_frame) contains the rules for restoring the historical rbp from rsp.

Red Zone (x86-64 System V)

The 128 bytes below rsp constitute the red zone—a function can use this area without adjusting rsp. This is valid only if the function does not call other functions (leaf function). Win64 has no red zone—on Windows, the area below rsp is not guaranteed to be safe from signal handlers or interrupts.

Prologue / Epilogue

Every function inserts standard sequences at the entry (prologue) and exit (epilogue):

Prologue:
    push rbp              ; Save frame pointer
    mov rbp, rsp          ; Set new frame pointer
    sub rsp, N            ; Allocate stack space for locals + spills + callee-saved regs
    push rbx (etc.)       ; Save callee-saved registers (if used by this function)

Epilogue:
    pop rbx (etc.)        ; Restore callee-saved registers
    mov rsp, rbp          ; Release stack space
    pop rbp               ; Restore old frame pointer
    ret

The compiler knows which registers are used by the current function—if rbx is not used, it won't push rbx. The frame layout (which register is at which stack position) is determined at compile time, and the push/pop operations and offsets in the prologue/epilogue are constants.

Variadic Functions

The number and types of arguments for printf(fmt, a, b, c, ...) are unknown at compile time. The x86-64 ABI specifies that variadic function argument passing is the same as normal functions (first 6 to registers, rest on stack), but the al register records how many XMM registers were used for floating-point arguments—this is key for printf internally to retrieve varargs.

Compiler requirements: At the vararg call site, the compiler must ensure that any arguments potentially passed in XMM registers are stored in the corresponding va_list locations. The implementation of va_start/va_arg is entirely based on the ABI's register and stack argument layout.

PIC / PIE: Position-Independent Code

Shared libraries and modern executables require PIC (Position-Independent Code): the code segment can be loaded at any memory location without needing relocation to modify the code itself (since the code segment is shared, modifications would affect all processes). The cost is an extra level of indirection when accessing global variables and functions:

Non-PIC:                           PIC:
  mov rax, [addr_of_x]              mov rax, [rip + offset_to_GOT_entry]
  call fn                           call [rip + offset_to_GOT_entry_for_fn]
                                     ; GOT = Global Offset Table, located in the data segment

When generating PIC, the compiler accesses all "addresses of external symbols" indirectly via the GOT—dynamic linkers fill GOT entries at load time. rip + offset is PC-relative addressing (no absolute addresses needed), ensuring the code segment is address-independent.

References

System V AMD64 ABI: https://gitlab.com/x86-psABIs/x86-64-ABI — Authoritative specification
Microsoft x64 ABI: https://learn.microsoft.com/en-us/cpp/build/x64-software-conventions
Agner Fog: "Calling Conventions" — Engineering summary of ABIs across platforms

Keywords: ABI, calling convention, System V, Win64, stack frame, prologue, epilogue, frame pointer, red zone, caller-saved, callee-saved, register passing, shadow space, variadic, va_list, AL register, PIC, PIE, GOT, PLT, PC-relative addressing, stack alignment