ftrace and bpftrace User-Space Tracing

Coverage: uprobe → USDT (DTrace probes) → bpftrace user-space scripts → perf probe on userspace → User-space dynamic tracing panorama Applicable to: Linux + bpftrace/bcc/perf

uprobe: User-Space Function Probes

# uprobe: Place probes at the entry/return of any user-space function
# Syntax: uprobe:<binary>:<function>

# Example: Trace readline in bash:
echo 'p:bash_readline /bin/bash:readline' \
     > /sys/kernel/debug/tracing/uprobe_events
echo 1 > /sys/kernel/debug/tracing/events/uprobes/bash_readline/enable
cat /sys/kernel/debug/tracing/trace_pipe
# → Each time bash calls readline → record

# With arguments:
echo 'p:bash_readline /bin/bash:readline cmd=+0(%si):string' \
     > uprobe_events
# cmd: The first argument after the function returns (%si=x0 on ARM, %rdi on x86)

# uretprobe: On function return
echo 'r:bash_readline_ret /bin/bash:readline ret=$retval' \
     > uprobe_events

USDT: Pre-planted Static Probes

// USDT (User Statically Defined Tracing): Similar to DTrace probes
// Planted at compile time, zero overhead (NOP) at runtime → until activated

#include <sys/sdt.h>  // systemtap-sdt-dev (Debian) or dev-util/systemtap (Gentoo)

DTRACE_PROBE2(myapp, request_start, request_id, url);
// → Compiled into a NOP sled → Replaced with INT3 breakpoint by uprobe at runtime

// View USDT probes embedded in the binary:
bpftrace -l 'usdt:/path/to/binary:*'
readelf -n /path/to/binary | grep NT_STAPSDT

bpftrace: User-Space Tracing

# One-line trace:
bpftrace -e 'uprobe:/bin/bash:readline { printf("readline: %s\n", str(reg("di"))); }'

# Histogram:
bpftrace -e 'uprobe:/lib64/libc.so.6:malloc { @size = hist(arg0); }'

# Statistics:
bpftrace -e 'uretprobe:/lib64/libc.so.6:malloc /
             retval != 0/ { @bytes[comm] = sum(arg0); }'

# By process:
bpftrace -e 'uprobe:/usr/bin/nginx:ngx_http_process_request {
             @requests[pid] = count(); }'

# Trace function latency:
bpftrace -e 'uprobe:/lib64/libc.so.6:malloc { @start[tid] = nsecs; }
             uretprobe:/lib64/libc.so.6:malloc /@start[tid]/ {
               @lat_us = hist((nsecs - @start[tid]) / 1000);
               delete(@start[tid]);
             }'

# Trace open calls (with filename):
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
             printf("%s %s\n", comm, str(args->filename)); }'

bcc User-Space Tools

# bcc: High-level wrapper of Python + BPF
# Pre-compiled tools (located in /usr/share/bcc/tools/):

execsnoop        # exec() family: Who forked whom
opensnoop        # open() tracing: filename, return value, latency
bitesize         # IO size statistics by process (histogram)
filelife         # Lifecycle of temporary files
statsnoop        # stat() tracing

# Use case: No need to write BPF programs → Use pre-compiled tools directly

Combination with Kernel Tracing

# Complete chain from user-space to kernel-space:
bpftrace -e '
uprobe:/lib64/libpthread.so.0:pthread_mutex_lock {
    @lock_start[tid] = nsecs;
}
uretprobe:/lib64/libpthread.so.0:pthread_mutex_lock {
    $duration = (nsecs - @lock_start[tid]) / 1000;
    if ($duration > 10000) {  // > 10μs → futex_wait may have occurred
        printf("slow lock: %s %d μs\n", comm, $duration);
    }
    delete(@lock_start[tid]);
}
tracepoint:syscalls:sys_enter_futex {
    // The tid here can be correlated with the above → confirm if futex caused the slowness
}'

Debugging Tips

# Find functions in a binary that can be probed:
nm -D /usr/lib64/libc.so.6 | grep ' T ' | head -20   # Exported symbols
objdump -T /usr/lib64/libc.so.6 | grep -E '\.text'    # Code section symbols

# View installed uprobes:
cat /sys/kernel/debug/tracing/uprobe_events
cat /sys/kernel/debug/tracing/kprobe_events

# View perf probes:
perf probe -l

# Cleanup:
echo > /sys/kernel/debug/tracing/uprobe_events

References

• Tools: bpftrace (github.com/bpftrace), bcc (github.com/iovisor/bcc)
• Documentation: man perf-probe, Brendan Gregg's blog
• LWN: "Uprobes and USDT", "Dynamic tracing with bpftrace"

Keywords: uprobe, uretprobe, USDT, DTRACE_PROBE, bpftrace, bcc, user-space tracing, SDT