---
title: perf 深度使用
url: https://doc.liz6.com/systems-programming/08-performance-and-debugging/01-deep-dive-into-perf
locale: zh
area: systems-programming
tags:
- systems-programming
- 性能与调试
date: 2026-06-30
modified: 2026-07-11
description: '覆盖: perf record/report/annotate → 火焰图 → perf stat → PEBS → perf probe → 性能分析方法论 适用: Linux perf_events (kernel 2.6.31+)'
---

# perf 深度使用

> 覆盖: perf record/report/annotate → 火焰图 → perf stat → PEBS → perf probe → 性能分析方法论
> 适用: Linux perf_events (kernel 2.6.31+)

## perf stat: CPU 性能计数器

```bash
# 计数型: 整个程序执行期间的硬件事件
perf stat ls /

# 输出:
#   task-clock:        # 实际 CPU 时间
#   cycles:            # CPU 周期 (可被频率缩放影响, 用 cycles:u 限制用户态)
#   instructions:      # 执行的指令数
#   IPC:               # instructions/cycles (>1 = 超标量利用率好)
#   branches:          # 分支指令
#   branch-misses:     # 分支预测失败
#   cache-misses:      # 缓存失效
#   context-switches:  # 上下文切换
#   page-faults:       # 页故障

# 关键指标:
#   IPC < 0.5: 严重 stalled (cache miss, branch miss, data dependency)
#   IPC > 2:   超标量高度利用
#   branch-miss rate > 5%: 分支预测问题
#   cache-miss rate > 10%: 数据局部性问题

# 指定事件:
perf stat -e cycles,instructions,cache-references,cache-misses,branch-misses ./prog
```

## perf record: 采样

```bash
# 采样 (默认 cycles, 采样率 4000 Hz):
perf record -g ./prog        # -g: 记录调用图 (callchain)
perf record -F 99 -g ./prog  # 99 Hz 采样 (类似火焰图标准)

# 报告:
perf report                   # 交互式 (函数/调用链)
perf report --sort=dso,sym    # 按 .so + 函数 排序
perf report -n --stdio        # 纯文本 + 采样数

# Annotate (混合源码+汇编+采样):
perf annotate function_name

# 实时 top:
perf top -e cycles
```

## 火焰图

```bash
# 1. 采样:
perf record -F 99 -g -- ./prog

# 2. 生成火焰图 (Brendan Gregg 工具):
perf script > out.perf
stackcollapse-perf.pl out.perf > out.folded
flamegraph.pl out.folded > flamegraph.svg

# 或一行:
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg
```

## perf probe: 动态探测

```bash
# 在任意用户态函数加探测点:
perf probe -x ./prog my_function
perf probe -x ./prog 'my_function arg1=%di arg2=%si'   # 捕获参数

# 在任意行加探测点:
perf probe -x ./prog my_file.c:42

# 记录:
perf record -e probe_myprog:my_function -g -- ./prog

# 查看已加探测点:
perf probe -l

# 删除:
perf probe -d my_function
```

## PEBS: 精确采样

```bash
# 默认采样: 中断后读取指令指针 (可能不精确, skid)
# PEBS (Precise Event-Based Sampling): 硬件记录精确 IP
perf record -e cycles:pp ./prog    # :pp = precise (2 level)
perf record -e cycles:ppp ./prog   # :ppp = most precise (需要硬件支持)
```

## 系统级分析

```bash
# 全系统采样 (需要 root):
perf record -a -g -- sleep 10     # 所有 CPU, 10 秒

# 按进程过滤:
perf record -e cycles -p <pid> -- sleep 10
perf record -e cycles -t <tid>    # 按线程

# 按 CPU 过滤:
perf record -e cycles -C 0,2 -- sleep 10

# CPU 利用率分解 (topdown):
perf stat --topdown ./prog        # Intel 6th gen+
# frontend bound / backend bound / bad speculation / retiring
```

## 性能分析方法论

<svg viewBox="0 0 720 390" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,'Source Han Sans CN','Microsoft YaHei',sans-serif" role="img" aria-label="性能分析方法论决策树:先怀疑瓶颈类型,再用对应 perf 命令定位">
  <defs>
    <marker id="pfah" markerWidth="10" markerHeight="8" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6 Z" fill="#475569"/></marker>
  </defs>
  <rect width="720" height="390" fill="#ffffff"/>
  <text x="360" y="28" text-anchor="middle" font-size="17" font-weight="700" fill="#1f2933">性能分析方法论:先定性瓶颈,再逐层深挖</text>

  <text x="95" y="48" text-anchor="middle" font-size="11" font-weight="600" fill="#64748b">怀疑什么</text>
  <text x="315" y="48" text-anchor="middle" font-size="11" font-weight="600" fill="#64748b">先用什么定性</text>
  <text x="580" y="48" text-anchor="middle" font-size="11" font-weight="600" fill="#64748b">再用什么深挖</text>

  <!-- row 1: CPU bound -->
  <rect x="20" y="58" width="150" height="40" rx="8" fill="#4f46e5"/>
  <text x="95" y="83" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">CPU bound?</text>
  <line x1="170" y1="78" x2="196" y2="78" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="200" y="58" width="230" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="212" y="83" font-size="11" fill="#115e59">perf stat: IPC / instructions / cycles</text>
  <line x1="430" y1="78" x2="456" y2="78" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="460" y="58" width="240" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="472" y="83" font-size="11" fill="#115e59">perf record: 找热点函数</text>

  <!-- row 2: memory bound -->
  <rect x="20" y="114" width="150" height="40" rx="8" fill="#4f46e5"/>
  <text x="95" y="139" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">memory bound?</text>
  <line x1="170" y1="134" x2="196" y2="134" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="200" y="114" width="230" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="212" y="134" font-size="11" fill="#115e59">perf stat: cache-misses,</text>
  <text x="212" y="147" font-size="11" fill="#115e59">LLC-loads / LLC-load-misses</text>
  <line x1="430" y1="134" x2="456" y2="134" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="460" y="114" width="240" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="472" y="139" font-size="11" fill="#115e59">perf record -e cache-misses</text>

  <!-- row 3: IO bound -->
  <rect x="20" y="170" width="150" height="40" rx="8" fill="#4f46e5"/>
  <text x="95" y="195" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">IO bound?</text>
  <line x1="170" y1="190" x2="196" y2="190" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="200" y="170" width="230" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="212" y="195" font-size="11" fill="#115e59">iostat / blktrace</text>
  <line x1="430" y1="190" x2="456" y2="190" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="460" y="170" width="240" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="472" y="195" font-size="11" fill="#115e59">perf trace(追踪系统调用+耗时)</text>

  <!-- row 4: 锁竞争 -->
  <rect x="20" y="226" width="150" height="40" rx="8" fill="#4f46e5"/>
  <text x="95" y="251" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">锁竞争?</text>
  <line x1="170" y1="246" x2="196" y2="246" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="200" y="226" width="230" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="212" y="251" font-size="11" fill="#115e59">perf lock record + perf lock report</text>
  <line x1="430" y1="246" x2="456" y2="246" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="460" y="226" width="240" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="472" y="251" font-size="11" fill="#115e59">perf stat -e context-switches</text>

  <!-- row 5: 分支预测 -->
  <rect x="20" y="282" width="150" height="40" rx="8" fill="#4f46e5"/>
  <text x="95" y="307" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">分支预测?</text>
  <line x1="170" y1="302" x2="196" y2="302" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="200" y="282" width="230" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="212" y="307" font-size="11" fill="#115e59">perf stat -e branch-misses,branches</text>
  <line x1="430" y1="302" x2="456" y2="302" stroke="#475569" stroke-width="1.6" marker-end="url(#pfah)"/>
  <rect x="460" y="282" width="240" height="40" rx="6" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="472" y="307" font-size="11" fill="#115e59">perf record -e branch-misses</text>

  <!-- insight -->
  <rect x="20" y="336" width="680" height="36" rx="8" fill="#eef2ff" stroke="#c7d2fe"/>
  <text x="360" y="358" text-anchor="middle" font-size="12" fill="#3730a3">先用 perf stat 定性瓶颈类型,再用 perf record / trace / lock 逐层深挖到具体函数或调用点。</text>
</svg>

## 参考

- **文档**: https://perf.wiki.kernel.org, Brendan Gregg's perf examples
- **工具**: https://github.com/brendangregg/FlameGraph
- **书籍**: "Systems Performance" (Brendan Gregg), "BPF Performance Tools"

*关键词: perf, PEBS, IPC, cache-misses, flame graph, perf probe, topdown, branch prediction*