---
title: ftrace and bpftrace User-Space Tracing
url: https://doc.liz6.com/en/systems-programming/08-performance-and-debugging/03-ftrace-and-bpftrace-user-space-tracing
locale: en
area: systems-programming
tags:
- systems-programming
- performance-and-debugging
date: 2026-06-30
modified: 2026-07-16
description: 'Coverage: uprobe → USDT (DTrace probes) → bpftrace user-space scripts → perf probe on userspace → User-space dynamic tracing panorama Applicable to: Linux + bpftrace/bcc/perf'
---

# ftrace and bpftrace User-Space Tracing

> Coverage: uprobe → USDT (DTrace probes) → bpftrace user-space scripts → perf probe on userspace → User-space dynamic tracing panorama
> Applicable to: Linux + bpftrace/bcc/perf

## uprobe: User-Space Function Probes

```bash
# uprobe: Place probes at the entry/return of any user-space function
# Syntax: uprobe:<binary>:<function>

# Example: Trace readline in bash:
echo 'p:bash_readline /bin/bash:readline' \
     > /sys/kernel/debug/tracing/uprobe_events
echo 1 > /sys/kernel/debug/tracing/events/uprobes/bash_readline/enable
cat /sys/kernel/debug/tracing/trace_pipe
# → Each time bash calls readline → record

# With arguments:
echo 'p:bash_readline /bin/bash:readline cmd=+0(%si):string' \
     > uprobe_events
# cmd: The first argument after the function returns (%si=x0 on ARM, %rdi on x86)

# uretprobe: On function return
echo 'r:bash_readline_ret /bin/bash:readline ret=$retval' \
     > uprobe_events
```

## USDT: Pre-planted Static Probes

```c
// USDT (User Statically Defined Tracing): Similar to DTrace probes
// Planted at compile time, zero overhead (NOP) at runtime → until activated

#include <sys/sdt.h>  // systemtap-sdt-dev (Debian) or dev-util/systemtap (Gentoo)

DTRACE_PROBE2(myapp, request_start, request_id, url);
// → Compiled into a NOP sled → Replaced with INT3 breakpoint by uprobe at runtime

// View USDT probes embedded in the binary:
bpftrace -l 'usdt:/path/to/binary:*'
readelf -n /path/to/binary | grep NT_STAPSDT
```

## bpftrace: User-Space Tracing

```bash
# One-line trace:
bpftrace -e 'uprobe:/bin/bash:readline { printf("readline: %s\n", str(reg("di"))); }'

# Histogram:
bpftrace -e 'uprobe:/lib64/libc.so.6:malloc { @size = hist(arg0); }'

# Statistics:
bpftrace -e 'uretprobe:/lib64/libc.so.6:malloc /
             retval != 0/ { @bytes[comm] = sum(arg0); }'

# By process:
bpftrace -e 'uprobe:/usr/bin/nginx:ngx_http_process_request {
             @requests[pid] = count(); }'

# Trace function latency:
bpftrace -e 'uprobe:/lib64/libc.so.6:malloc { @start[tid] = nsecs; }
             uretprobe:/lib64/libc.so.6:malloc /@start[tid]/ {
               @lat_us = hist((nsecs - @start[tid]) / 1000);
               delete(@start[tid]);
             }'

# Trace open calls (with filename):
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
             printf("%s %s\n", comm, str(args->filename)); }'
```

## bcc User-Space Tools

```bash
# bcc: High-level wrapper of Python + BPF
# Pre-compiled tools (located in /usr/share/bcc/tools/):

execsnoop        # exec() family: Who forked whom
opensnoop        # open() tracing: filename, return value, latency
bitesize         # IO size statistics by process (histogram)
filelife         # Lifecycle of temporary files
statsnoop        # stat() tracing

# Use case: No need to write BPF programs → Use pre-compiled tools directly
```

## Combination with Kernel Tracing

```bash
# Complete chain from user-space to kernel-space:
bpftrace -e '
uprobe:/lib64/libpthread.so.0:pthread_mutex_lock {
    @lock_start[tid] = nsecs;
}
uretprobe:/lib64/libpthread.so.0:pthread_mutex_lock {
    $duration = (nsecs - @lock_start[tid]) / 1000;
    if ($duration > 10000) {  // > 10μs → futex_wait may have occurred
        printf("slow lock: %s %d μs\n", comm, $duration);
    }
    delete(@lock_start[tid]);
}
tracepoint:syscalls:sys_enter_futex {
    // The tid here can be correlated with the above → confirm if futex caused the slowness
}'
```

## Debugging Tips

```bash
# Find functions in a binary that can be probed:
nm -D /usr/lib64/libc.so.6 | grep ' T ' | head -20   # Exported symbols
objdump -T /usr/lib64/libc.so.6 | grep -E '\.text'    # Code section symbols

# View installed uprobes:
cat /sys/kernel/debug/tracing/uprobe_events
cat /sys/kernel/debug/tracing/kprobe_events

# View perf probes:
perf probe -l

# Cleanup:
echo > /sys/kernel/debug/tracing/uprobe_events
```

## References

- **Tools**: bpftrace (github.com/bpftrace), bcc (github.com/iovisor/bcc)
- **Documentation**: `man perf-probe`, Brendan Gregg's blog
- **LWN**: "Uprobes and USDT", "Dynamic tracing with bpftrace"

*Keywords: uprobe, uretprobe, USDT, DTRACE_PROBE, bpftrace, bcc, user-space tracing, SDT*