Anatomy of glibc Syscall Wrappers

Coverage: glibc syscall wrapper → INLINE_SYSCALL → vsyscall/vDSO → gettimeofday fast path → syscall latency → custom syscall wrapper Applicable to: glibc 2.x, with comparisons to musl, primarily on x86-64

Overview

User-space code does not directly emit the syscall instruction; glibc wraps every system call with a layer of abstraction. This wrapper layer handles: argument passing (register allocation), return value to errno conversion, cancellation points, and the vDSO fast path. Understanding this layer explains why "clock_gettime is faster than getpid" or why "a custom syscall wrapper can sometimes be faster."

glibc: The INLINE_SYSCALL Macro

// sysdeps/unix/sysv/linux/x86_64/sysdep.h
// glibc syscall wrapper hierarchy:

// Lowest level: Assembly macro
#define INTERNAL_SYSCALL(name, nr, args...)                \
({                                                          \
    unsigned long int resultvar;                            \
    LOAD_ARGS_##nr(args)          /* Place arguments into registers */ \
    LOAD_REGS_##nr                 /* Load rdi, rsi, rdx, ... */ \
    register long int _a1 asm ("rdi");                      \
    asm volatile (                                          \
        "syscall\n\t"                                       \
        : "=a" (resultvar)                                  \
        : "0" (__NR_##name) ASM_ARGS_##nr                   \
        : "memory", "cc", "r11", "cx");                     \
    (long int) resultvar;                                   \
})

// Application layer: Convert to errno
#define INLINE_SYSCALL(name, nr, args...)                  \
({                                                          \
    long int sc_ret = INTERNAL_SYSCALL(name, nr, args);    \
    if (__glibc_unlikely(INTERNAL_SYSCALL_ERROR_P(sc_ret))) \
    {                                                       \
        __set_errno(INTERNAL_SYSCALL_ERRNO(sc_ret));       \
        sc_ret = -1;                                        \
    }                                                       \
    sc_ret;                                                 \
})

Key details:

• asm volatile: Prevents the compiler from reordering memory accesses across the syscall (via the "memory" clobber).
• "cc" clobber: The syscall instruction may modify the flags register.
• "r11", "cx" clobber: The syscall instruction destroys these two registers.
• Return value check: Uses __glibc_unlikely branch prediction to indicate that the errno path is rarely taken.

gettimeofday / clock_gettime: vDSO Fast Path

// glibc's clock_gettime implementation (simplified):
int __clock_gettime64(clockid_t clock_id, struct __timespec64 *tp)
{
    // Attempt vDSO fast path (executes in user space, no syscall)
    if (__glibc_likely(clock_id < CLOCK_THREAD_CPUTIME_ID))
    {
        __vdso_clock_gettime_func vdso_func =
            GLRO(dl_vdso_clock_gettime);  // Obtained from vDSO at startup

        if (vdso_func != NULL)
        {
            int ret = vdso_func(clock_id, tp);
            if (ret == 0) return 0;
        }
    }
    // vDSO unavailable → fallback to syscall
    return INLINE_SYSCALL(clock_gettime64, 2, clock_id, tp);
}

// vDSO version (arch/x86/entry/vdso/vclock_gettime.c):
//   1. Read TSC (rdtsc, ~10 cycles)
//   2. Apply timekeeper conversion (mult + shift, ~20 cycles)
//   3. Check seqlock (read_seqbegin/read_seqretry, ~5 cycles)
//   → Total ~35 cycles, vs ~70 cycles for syscall → ~2x faster

glibc vs musl: Syscall Wrapper Differences

Syscall Latency Analysis

# Measure syscall latency (minimum 0-arg syscall):
# getpid() → syscall 39 (x86-64, getpid)
#   PID is cached → returns directly → does not actually read the PCB

# Syscall latency measurement tools: libMicro, lmbench

Typical Latency (3.5GHz x86-64):
  getpid (cached):                ~70ns (~250 cycles)
  getppid:                        ~90ns
  gettimeofday (vDSO):            ~15ns (~50 cycles)  ← Extremely fast!
  gettimeofday (syscall fallback): ~80ns
  write(1 byte to /dev/null):     ~200ns
  read(1 byte from /dev/zero):    ~200ns
  futex(FUTEX_WAIT, uncontended): ~150ns
  futex(FUTEX_WAIT, contended):   ~5-20μs (including context switch)

Latency Breakdown:
  syscall instruction:            ~20 cycles
  Kernel entry/exit (swapgs + stack switch): ~20 cycles
  Actual syscall logic:           ~30+ cycles
  Spectre/Meltdown Mitigations:   ~20 cycles (KPTI on affected CPUs)

Hand-written Syscall: A Faster Wrapper

// glibc's read() has some overhead (errno conversion, etc.)
// If you are on a hot path and can handle errors yourself → write the syscall manually:

static inline long __read(int fd, void *buf, size_t count) {
    long ret;
    register long rdi asm("rdi") = fd;
    register long rsi asm("rsi") = (long)buf;
    register long rdx asm("rdx") = count;
    register long rax asm("rax") = __NR_read;
    asm volatile("syscall"
                 : "=a"(ret)
                 : "r"(rdi), "r"(rsi), "r"(rdx), "a"(rax)
                 : "rcx", "r11", "memory");
    return ret;  // Negative value = -errno
}

// When is it worth it?
//   - io_uring hot path → already wrapped, not needed
//   - Custom allocator → might need an mmap wrapper
//   - Performance benchmarking → to exclude glibc overhead

References

• Source Code: glibc sysdeps/unix/sysv/linux/x86_64/sysdep.h, sysdeps/unix/sysv/linux/clock_gettime.c, musl arch/x86_64/syscall_arch.h
• LWN: "vDSO and clock_gettime", "System call overhead"
• Tools: man syscalls, strace -c (syscall count/time)

Keywords: INLINE_SYSCALL, syscall wrapper, vDSO, clock_gettime, errno, cancellation point, glibc, musl, syscall latency