---
title: 追踪与调试
url: https://doc.liz6.com/linux-kernel/08-ebpf-and-observability/03-tracing-and-debugging
locale: zh
area: linux-kernel
tags:
- linux-kernel
- eBPF与可观测性
date: 2026-06-30
modified: 2026-06-30
description: '覆盖: ftrace (function/function_graph) → tracepoints → kprobes → perf events → eBPF 追踪 (bpftrace/bcc) → kdump/crash → KGDB → printk 与动态调试 内核版本: 2.6 ~ 6.x'
---

# 追踪与调试

> 覆盖: ftrace (function/function_graph) → tracepoints → kprobes → perf events → eBPF 追踪 (bpftrace/bcc) → kdump/crash → KGDB → printk 与动态调试
> 内核版本: 2.6 ~ 6.x

## ftrace: 内核内建追踪器

### Function Tracer

```bash
# 列出可用 tracer
cat /sys/kernel/debug/tracing/available_tracers

# function: 追踪所有函数调用 (极高开销, 仅调试用)
echo function > /sys/kernel/debug/tracing/current_tracer
echo "tcp_v4_rcv" > /sys/kernel/debug/tracing/set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace_pipe
```

### Function Graph: 带耗时

```bash
echo function_graph > /sys/kernel/debug/tracing/current_tracer
# 输出:
# 0)               |  tcp_v4_rcv() {
# 0)   0.234 us    |    tcp_checksum_complete();
# 0)   0.567 us    |    tcp_v4_do_rcv() {
# 0)   1.234 us    |      tcp_rcv_established();
# 0)   2.345 us    |    }
# 0)   3.456 us    |  }
```

### 动态 ftrace: 运行时开启/关闭

```bash
# 动态函数探测 (通过 debugfs):
echo 'p:tcp_probe tcp_rcv_established state+0x12(%di):u8' \
     > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/tcp_probe/enable
cat /sys/kernel/debug/tracing/trace_pipe
```

---

## Tracepoints: 稳定探测点

```bash
# 所有可用 tracepoint
ls /sys/kernel/debug/tracing/events/
# sched/  syscalls/  net/  block/  irq/  timer/  ...

# 追踪所有 exec 调用:
echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_execve/enable
cat trace_pipe

# 追踪调度器:
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable

# 追踪块 IO:
echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_issue/enable
echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_complete/enable
```

---

## Perf Events: 硬件性能计数器

```bash
# perf stat: 统计程序执行期间的硬件计数
perf stat -e cycles,instructions,cache-misses,branch-misses -- ls /

# perf record: 采样 (生成火焰图的基础)
perf record -e cycles -g -- sleep 10
perf report

# perf top: 实时热点
perf top -e cycles

# 硬件事件:
#   cycles, instructions, cache-references, cache-misses,
#   branch-instructions, branch-misses, bus-cycles,
#   stalled-cycles-frontend, stalled-cycles-backend
```

---

## eBPF 追踪: bpftrace

```bash
# bpftrace: DTrace 风格的高级追踪语言
# 单行追踪:
bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# 直方图:
bpftrace -e 'kprobe:vfs_read { @usecs[comm] = hist(nsecs / 1000); }'
bpftrace -e 'kprobe:submit_bio { @latency = hist(nsecs); @count = count(); }'

# 聚合:
bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @bytes[comm] = sum(retval); }'
```

### bcc 工具集

```bash
# bcc: Python + BPF 的预编译工具集 (/usr/share/bcc/tools/)
execsnoop        # 追踪所有 exec()
opensnoop        # 追踪所有 open()
biolatency       # 块 IO 延迟直方图 (按设备/进程)
tcptop           # TCP 流量的实时 top
tcpretrans       # TCP 重传统计
cachestat        # page cache 命中率
runqlat          # 调度器延迟 (等 CPU 的时间)
```

---

## printk 与动态调试

### printk 日志等级

```c
// include/linux/kern_levels.h
#define KERN_EMERG   "<0>"  // 系统崩溃
#define KERN_ALERT   "<1>"  // 立即行动
#define KERN_CRIT    "<2>"  // 严重错误
#define KERN_ERR     "<3>"  // 错误
#define KERN_WARNING "<4>"  // 警告
#define KERN_NOTICE  "<5>"  // 正常但重要
#define KERN_INFO    "<6>"  // 信息
#define KERN_DEBUG   "<7>"  // 调试

// 运行时控制控制台日志等级:
echo 4 > /proc/sys/kernel/printk  // 只显示 ERROR 及以上
```

### Dynamic Debug (dyndbg)

```bash
# 运行时开启特定模块的调试输出 (不需要重编)
echo "module nfs file fs/nfs/* +p" > /sys/kernel/debug/dynamic_debug/control
echo "func tcp_rcv_established +p" > /sys/kernel/debug/dynamic_debug/control

# 查看当前启用的动态调试条目
cat /sys/kernel/debug/dynamic_debug/control | grep "=p"
```

---

## kdump: 崩溃转储

```bash
# 1. 加载捕获内核 (启动时或运行时)
kexec -p /boot/vmlinuz-capture --initrd=/boot/initrd-capture \
      --command-line="root=... irqpoll nr_cpus=1 reset_devices"

# 2. 触发 panic → 捕获内核启动 → vmcore 生成
# 3. 分析:
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/vmcore

# crash 命令:
crash> bt          # 所有进程的内核栈
crash> log         # panic 时的 printk 缓冲
crash> ps          # panic 时的进程列表
crash> foreach bt  # 所有任务的栈回溯
```

---

## KGDB: 内核调试器

```
通过串口远程调试内核 (类似 gdb server):
  → 设断点, 单步执行, 读寄存器, 读内存
  → 但只能用于开发/调试环境 (需要特殊配置)

启用: kgdboc=ttyS0,115200 kgdbwait (boot param)
连接: gdb vmlinux → target remote /dev/ttyS0
```

---

## 实践: 性能问题诊断步骤

```
1. 用 perf top 看 CPU 热点
2. 用 bpftrace/bcc 看延迟分布 (biolatency, runqlat, tcpretrans)
3. 用 ftrace function_graph 追踪关键路径的调用耗时
4. 用 tracepoint 验证假设 (如 sched_switch 频率过高?)
5. 用 eBPF 写自定义工具 (如果有必要)
```

---

## 参考

- **源码**: `kernel/trace/`, `kernel/kexec_core.c`, `kernel/printk/`
- **书籍**: "BPF Performance Tools" (Brendan Gregg), "Systems Performance" (同作者)
- **工具**: perf, bpftrace, bcc, crash, trace-cmd (ftrace 前端)

---

*关键词: ftrace, function_graph, tracepoint, kprobe, perf, bpftrace, bcc, kdump, crash, dynamic debug, KGDB*