Object Code and ABI

An ABI (Application Binary Interface) is a convention between binary components, covering data layout, calling conventions, symbol and object formats, and more. Even if the code on either side of a function boundary is generated by different compilers, they must agree on where arguments are located, which registers may be modified, and how return values are represented.

Violating an ABI can cause an immediate crash, or it may silently read incorrect arguments or corrupt a value that is used later. This article uses common user-mode x86-64 calling conventions as examples; system calls, kernel code, and special calling conventions require separate treatment.

Classify Arguments First, Then Assign Locations

For the first six common integer or pointer arguments in System V AMD64, the registers RDI, RSI, RDX, RCX, R8, and R9 are used; floating-point arguments follow a separate set of XMM register rules. Aggregate types must be classified according to the ABI; you cannot simply stuff all arguments into the same "first six" list.

For Windows x64, the first four argument locations use RCX, RDX, R8, R9, or the corresponding XMM0–3 depending on their types. The caller reserves at least 32 bytes of register parameter home/shadow space. Mixed integer and floating-point arguments do not form two independent, unlimited queues of the "top four." Microsoft x64 Calling Convention

Return values are also classified by type: scalar integers typically use RAX, scalar floating-point typically uses XMM0, some aggregates can be split across multiple registers, and others are returned via a hidden result pointer. The statement "128-bit structures are uniformly placed in RDX:RAX" is incorrect. AMD64 ABI

Why Register Allocation Differs for the Same Mixed Argument Declaration

Consider the fixed prototype double mix(int a, double b, int c, double d), which does not involve variadic arguments or aggregates. System V consumes integer and floating-point argument sequences separately: a uses EDI, b uses XMM0, c uses ESI, and d uses XMM1. Windows x64 selects registers based on argument position: a uses ECX, b uses XMM1, c uses R8D, and d uses XMM3.

In the table, int only uses the corresponding 32-bit value; do not infer from the declaration that the remaining bits of the register can be used to pass other arguments. The following source file does not depend on platform headers and can generate assembly for both targets separately:

/* mix.c */
double mix(int a, double b, int c, double d) {
    return a + b + c + d;
}

clang --target=x86_64-unknown-linux-gnu -O2 -S -masm=intel mix.c -o mix-sysv.s
clang --target=x86_64-pc-windows-msvc -O2 -S -masm=intel mix.c -o mix-win.s

Instructions converting integers to double can identify the two integer sources, followed by observing the floating-point inputs along the addition chain. Here we only generate assembly across targets; there is no need to run the Windows program. The return value is XMM0 in both cases, but the input layout differs, and these differences cannot be eliminated by casting function pointers. Microsoft Parameter Passing

Why are positional slots not independent type queues?

Changing argument types recomputes locations. System V advances separate int/double sequences; Windows keeps positional slots, so a double in position2 uses XMM1 even when XMM0 was unused. Beyond position4 this example uses stack slots regardless of unused registers.

Saving Conventions: Who Ensures Values Survive Across Calls

"Caller-saved" means the callee may modify the register; if the caller still needs the old value, it must save it or move it to a different location first. "Callee-saved" means that if the callee uses that register, it must restore the caller's value before returning.

In System V, RBX, RBP, and R12–R15 are common callee-saved registers, and RSP must be restored according to the calling convention upon return. RAX, RCX, RDX, RSI, RDI, and R8–R11 may be destroyed by the call. Vector registers and status registers also have their own rules; do not check only general-purpose registers.

This directly affects register allocation: values that need to survive across calls cannot be placed unprotected in locations that may be destroyed by the call.

Stack Layout and Alignment

For common System V calls, RSP satisfies 16-byte alignment before the call. After the call pushes the 8-byte return address, (RSP+8) at the function entry satisfies this alignment. Some cases involving vectors passed on the stack have stricter requirements; refer to the corresponding ABI rules.

Therefore, when demonstrating seven integer arguments, you cannot simply write push 7th_arg; call while completely ignoring the current stack state and padding. The caller may reserve a block of argument space and then write into it; there is no need to push arguments one by one in source code order.

Local variables are not necessarily on the stack; after optimization, they may be in registers, replaced by constants, or disappear entirely. The frame pointer may also be omitted; unwind information describes how to restore the caller's state, not just a table for finding the old RBP from RSP. At runtime in ELF, unwinding often involves .eh_frame and other information.

Prologue and Epilogue: Code that Satisfies Conventions is Not a Fixed Template

The prologue may save callee-saved registers, allocate stack space, and set up the frame pointer; the epilogue restores the corresponding state. Leaf functions may not perform these actions, and optimization can delay saving until it is actually needed.

Below is a GNU assembly example calling an external twice(long) under Linux System V and adding 1. Using Intel syntax, the entry RSP modulo 16 is 8; subtracting 8 establishes alignment for the subsequent call.

.intel_syntax noprefix
.text
.globl plus_one_twice
.type plus_one_twice, @function
plus_one_twice:
    .cfi_startproc
    sub rsp, 8
    .cfi_def_cfa_offset 16
    call twice@PLT
    add rsp, 8
    .cfi_def_cfa_offset 8
    add rax, 1
    ret
    .cfi_endproc
.size plus_one_twice, .-plus_one_twice
.section .note.GNU-stack,"",@progbits

The first long argument is already in RDI, and the return value is in RAX; this function does not use registers like RBX that require extra saving. CFI describes the CFA (CFA is the Canonical Frame Address) along with stack adjustments, allowing the unwinder to restore the caller's state.

You Must Preserve Original Arguments if You Need Them After the Call

Change the calculation to twice(x) + x. x is initially in RDI, and twice is allowed to overwrite RDI, so adding RDI directly after return is incorrect. Below, we first save the caller's RBX to the stack, then use RBX to preserve x; twice must restore the RBX it received, and finally this function restores the upper caller's old value.

# keep_x.s: GNU Assembly, Linux System V AMD64
.intel_syntax noprefix
.text
.globl keep_x
.type keep_x, @function
keep_x:
    .cfi_startproc
    push rbx
    .cfi_def_cfa_offset 16
    .cfi_offset rbx, -16
    mov rbx, rdi
    call twice@PLT
    add rax, rbx
    pop rbx
    .cfi_def_cfa_offset 8
    .cfi_restore rbx
    ret
    .cfi_endproc
.size keep_x, .-keep_x
.section .note.GNU-stack,"",@progbits

Entry RSP modulo 16 is 8; the pushed 8 bytes both save RBX and make RSP modulo 16 equal to 0 before the call. You should not copy the sub rsp, 8 from the previous section and subtract again, or you will break the alignment at this call site.

/* abi-main.c: Compile together with keep_x.s */
extern long keep_x(long);
long twice(long x) { return x * 2; }
int main(void) {
    return keep_x(7) != 21 || keep_x(-3) != -9;
}

After executing cc -O2 abi-main.c keep_x.s -o abi-demo and running abi-demo, the normal exit status is 0. Using only these limited small values here avoids letting signed overflow in long multiplication or addition mask ABI issues.

A compiler happening not to modify RDI does not prove that the incorrect writing is correct. Checking call boundaries should be based on what state the callee is allowed to destroy, rather than just observing a single instance of machine code; implementation changes, instrumentation, or linking to another compatible definition may expose latent errors.

Where can x survive a call reliably?

Preserving x, restoring the caller’s register and aligning the call are independent checks. Construct a numeric21 that corrupts RBX, or correct values with misalignment from an extra stack subtraction. Numbers are symbolic evaluation of the example callee; after an ABI violation they are not guarantees of real machine behavior.

Red Zone: Temporary Values Cannot Cross Calls Unprotected

System V user-mode provides a 128-byte red zone below RSP, which must adhere to relevant conventions for signal handling, etc. Functions can use it to save temporary data without adjusting RSP; this is particularly common in leaf functions.

However, nested calls will use stack space at lower addresses, so you cannot expect values in the red zone to safely cross ordinary function calls. Non-leaf functions can also use temporary data that does not need to survive across calls in appropriate intervals, so the statement "only leaf functions can use it in any sense" is too absolute. Kernels, interrupts, and other platforms cannot directly copy this guarantee.

Variadic Arguments: The Caller Knows the Actual Arguments, the Callee Retrieves Them According to Convention

Declarations for functions like printf do not list all actual argument types, but the compiler at the call site knows the actual expressions and applies default argument promotions, such as promoting float to double. Whether the format string matches the actual arguments is still a matter of language and interface checking.

System V uses AL for variadic calls to indicate the upper bound of the number of vector register arguments used, with the value range defined by the ABI. The callee establishes register save areas as needed, and va_list tracks the register area and stack argument area. It is not the case that the caller universally pre-writes all XMM arguments into the callee's va_list.

Windows' variadic rules are different; floating-point actual arguments may need to be placed in general-purpose registers at relevant positions. Hand-written bridges across ABIs must handle this specifically and cannot simply replace argument register names.

PIC and PIE: Reducing Absolute Address Dependencies

Position-independent code often uses PC-relative addressing to access local data and code, and uses GOT, PLT, or other convention paths for symbols that can be dynamically bound. Not all PIC symbol accesses necessarily incur an extra GOT indirection: binding, visibility, and linker relaxation may allow direct relative access.

PIE (Position Independent Executable) is one form of position-independent executable, used in conjunction with load address randomization. Whether a certain relocation is needed and where it occurs should be combined with object format and symbol rules. See Linking and Loading.

Cross-Language Boundary Quick Reference

Check function prototypes, data size and alignment, aggregate classification, calling conventions, whether exceptions can cross boundaries, who frees memory, and symbol names. Declarations like extern "C" can specify some linking and calling rules, but they do not automatically make all language objects have compatible layouts and lifetimes.

References

• AMD64 System V ABI.
• Microsoft x64 Calling Convention.

Keywords: ABI, calling convention, stack alignment, caller-saved, callee-saved, shadow space, red zone, variadic, CFI, PIC, PIE