Deep Dive into perf

Coverage: perf record/report/annotate → Flame Graph → perf stat → PEBS → perf probe → Performance Analysis Methodology Applicable to: Linux perf_events (kernel 2.6.31+)

perf stat: CPU Performance Counters

# Counting mode: Hardware events during the entire program execution
perf stat ls /

# Output:
#   task-clock:        # Actual CPU time
#   cycles:            # CPU cycles (can be affected by frequency scaling; use cycles:u to restrict to user space)
#   instructions:      # Number of executed instructions
#   IPC:               # instructions/cycles (>1 = good superscalar utilization)
#   branches:          # Branch instructions
#   branch-misses:     # Branch prediction misses
#   cache-misses:      # Cache misses
#   context-switches:  # Context switches
#   page-faults:       # Page faults

# Key Metrics:
#   IPC < 0.5: Severe stalling (cache miss, branch miss, data dependency)
#   IPC > 2:   High superscalar utilization
#   branch-miss rate > 5%: Branch prediction issues
#   cache-miss rate > 10%: Data locality issues

# Specify events:
perf stat -e cycles,instructions,cache-references,cache-misses,branch-misses ./prog

perf record: Sampling

# Sampling (default cycles, sampling rate 4000 Hz):
perf record -g ./prog        # -g: Record call graph (callchain)
perf record -F 99 -g ./prog  # 99 Hz sampling (similar to standard flame graph)

# Reporting:
perf report                   # Interactive (functions/call chains)
perf report --sort=dso,sym    # Sort by .so + function
perf report -n --stdio        # Plain text + sample count

# Annotate (mixed source code + assembly + sampling):
perf annotate function_name

# Real-time top:
perf top -e cycles

Flame Graph

# 1. Sampling:
perf record -F 99 -g -- ./prog

# 2. Generate Flame Graph (Brendan Gregg's tools):
perf script > out.perf
stackcollapse-perf.pl out.perf > out.folded
flamegraph.pl out.folded > flamegraph.svg

# Or in one line:
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

perf probe: Dynamic Probing

# Add a probe point to any user-space function:
perf probe -x ./prog my_function
perf probe -x ./prog 'my_function arg1=%di arg2=%si'   # Capture arguments

# Add a probe point at any line:
perf probe -x ./prog my_file.c:42

# Record:
perf record -e probe_myprog:my_function -g -- ./prog

# View added probe points:
perf probe -l

# Delete:
perf probe -d my_function

PEBS: Precise Sampling

# Default sampling: Reads instruction pointer after interrupt (may be imprecise, skid)
# PEBS (Precise Event-Based Sampling): Hardware records precise IP
perf record -e cycles:pp ./prog    # :pp = precise (2 levels)
perf record -e cycles:ppp ./prog   # :ppp = most precise (requires hardware support)

System-Level Analysis

# System-wide sampling (requires root):
perf record -a -g -- sleep 10     # All CPUs, for 10 seconds

# Filter by process:
perf record -e cycles -p <pid> -- sleep 10
perf record -e cycles -t <tid>    # By thread

# Filter by CPU:
perf record -e cycles -C 0,2 -- sleep 10

# CPU utilization breakdown (topdown):
perf stat --topdown ./prog        # Intel 6th gen and later
# frontend bound / backend bound / bad speculation / retiring

Performance Analysis Methodology

Suspect What First Qualify With Then Deep Dive With