--- title: Deep Dive into perf url: https://doc.liz6.com/en/systems-programming/08-performance-and-debugging/01-deep-dive-into-perf locale: en area: systems-programming tags: - systems-programming - performance-and-debugging date: 2026-06-30 modified: 2026-07-16 description: 'Coverage: perf record/report/annotate → Flame Graph → perf stat → PEBS → perf probe → Performance Analysis Methodology Applicable to: Linux perf_events (kernel 2.6.31+)' --- # Deep Dive into perf > Coverage: perf record/report/annotate → Flame Graph → perf stat → PEBS → perf probe → Performance Analysis Methodology > Applicable to: Linux perf_events (kernel 2.6.31+) ## perf stat: CPU Performance Counters ```bash # Counting mode: Hardware events during the entire program execution perf stat ls / # Output: # task-clock: # Actual CPU time # cycles: # CPU cycles (can be affected by frequency scaling; use cycles:u to restrict to user space) # instructions: # Number of executed instructions # IPC: # instructions/cycles (>1 = good superscalar utilization) # branches: # Branch instructions # branch-misses: # Branch prediction misses # cache-misses: # Cache misses # context-switches: # Context switches # page-faults: # Page faults # Key Metrics: # IPC < 0.5: Severe stalling (cache miss, branch miss, data dependency) # IPC > 2: High superscalar utilization # branch-miss rate > 5%: Branch prediction issues # cache-miss rate > 10%: Data locality issues # Specify events: perf stat -e cycles,instructions,cache-references,cache-misses,branch-misses ./prog ``` ## perf record: Sampling ```bash # Sampling (default cycles, sampling rate 4000 Hz): perf record -g ./prog # -g: Record call graph (callchain) perf record -F 99 -g ./prog # 99 Hz sampling (similar to standard flame graph) # Reporting: perf report # Interactive (functions/call chains) perf report --sort=dso,sym # Sort by .so + function perf report -n --stdio # Plain text + sample count # Annotate (mixed source code + assembly + sampling): perf annotate function_name # Real-time top: perf top -e cycles ``` ## Flame Graph ```bash # 1. Sampling: perf record -F 99 -g -- ./prog # 2. Generate Flame Graph (Brendan Gregg's tools): perf script > out.perf stackcollapse-perf.pl out.perf > out.folded flamegraph.pl out.folded > flamegraph.svg # Or in one line: perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg ``` ## perf probe: Dynamic Probing ```bash # Add a probe point to any user-space function: perf probe -x ./prog my_function perf probe -x ./prog 'my_function arg1=%di arg2=%si' # Capture arguments # Add a probe point at any line: perf probe -x ./prog my_file.c:42 # Record: perf record -e probe_myprog:my_function -g -- ./prog # View added probe points: perf probe -l # Delete: perf probe -d my_function ``` ## PEBS: Precise Sampling ```bash # Default sampling: Reads instruction pointer after interrupt (may be imprecise, skid) # PEBS (Precise Event-Based Sampling): Hardware records precise IP perf record -e cycles:pp ./prog # :pp = precise (2 levels) perf record -e cycles:ppp ./prog # :ppp = most precise (requires hardware support) ``` ## System-Level Analysis ```bash # System-wide sampling (requires root): perf record -a -g -- sleep 10 # All CPUs, for 10 seconds # Filter by process: perf record -e cycles -p -- sleep 10 perf record -e cycles -t # By thread # Filter by CPU: perf record -e cycles -C 0,2 -- sleep 10 # CPU utilization breakdown (topdown): perf stat --topdown ./prog # Intel 6th gen and later # frontend bound / backend bound / bad speculation / retiring ``` ## Performance Analysis Methodology ## References - **Documentation**: https://perf.wiki.kernel.org, Brendan Gregg's perf examples - **Tools**: https://github.com/brendangregg/FlameGraph - **Books**: "Systems Performance" (Brendan Gregg), "BPF Performance Tools" *Keywords: perf, PEBS, IPC, cache-misses, flame graph, perf probe, topdown, branch prediction*