Deep Dive into perf

perf is the user-space tool for Linux perf_events. It can both count total events and sample call stacks when events occur. First, clarify the question you want to answer: How much CPU did the program consume? Where are the hotspots? Why is a specific instruction slow? Or what is the request waiting for? Different questions require different events and observation scopes.

The following commands use the Linux user-space program ./prog as an example. Event support depends on the CPU, kernel, virtualization environment, and perf version; use perf list to check what is actually available. Permissions also depend on perf_event_paranoid, capabilities, and system policies; not all operations require root.

Counting: Establish a Repeatable Baseline

perf stat -r 5 -e task-clock,context-switches,page-faults -- ./prog
perf stat -r 5 -e '{cycles:u,instructions:u}' -- ./prog
perf list

The first command starts with CPU time, scheduling, and page faults; the second groups user-space cycles and instructions into the same group for easier comparison within the same measurement window. -r 5 runs the program five times; input, data state, and warm-up strategies must be consistent. If the program modifies data, prepare a repeatable test scenario first.

For example, after rewriting an algorithm, instruction count drops by half, and IPC also drops, but total execution time might still be shorter. IPC should be interpreted in conjunction with execution time and completed work. The u in cycles:u means user-space only, not eliminating frequency changes; CPU frequency, SMT, migration, hybrid cores, and background interference still affect comparisons.

The specific mapping of generic cache-misses is determined by the PMU and should not be assumed to be the L1 miss rate. When discussing a specific cache level, select the event defined in that CPU's documentation and verify the denominator.

Same Work, Lower IPC Can Still Be Faster

The data below is hypothetical, intended for practicing counting interpretation, not measurement results from a specific machine. Both A and B complete 10,000 identical units of work, with consistent event scope and running conditions.

The correct conclusion here is that B completes the same work faster; do not judge it as a regression just because IPC dropped from 2 to 1.5. If B only processed 8,000 units, skipped validation, or swallowed errors, the reduced time cannot be directly counted as optimization benefit.

from math import isclose
jobs = 10_000
a = {"seconds": 0.5, "instructions": 2_000_000_000, "cycles": 1_000_000_000}
b = {"seconds": 0.4, "instructions": 1_200_000_000, "cycles": 800_000_000}
assert a["instructions"] / a["cycles"] == 2
assert b["instructions"] / b["cycles"] == 1.5
assert a["instructions"] / jobs == 200_000
assert b["instructions"] / jobs == 120_000
assert isclose(1 - b["seconds"] / a["seconds"], 0.2)
assert isclose(jobs / b["seconds"] / (jobs / a["seconds"]), 1.25)

A 20% reduction in time corresponds to a 25% increase in throughput, because their denominators differ. task-clock also requires the correct denominator: if 3.2 seconds of cumulative CPU time are recorded within 2 seconds of wall-clock time, the average is approximately 1.6 CPUs of execution time, not "CPU utilization exceeded 100% so the data is broken." This average still does not represent peak usage at any moment or uniformity across cores.

Counter Multiplexing: Numbers May Be Scaled

Hardware counters available simultaneously are limited. When events exceed available resources, they may be time-multiplexed. perf can scale results based on actual running time, but if the load is highly phased, briefly observing one phase and extrapolating to the whole process introduces error.

Pay attention to the running ratio, not counted, or not supported in the output. Reducing the number of events, grouping related ratio events, and repeating experiments help judge result stability; grouping may also fail to schedule entirely due to resource constraints. See the manual for perf stat scaling and event group semantics.

Scaling Correction Covers Time, Cannot Recover Unobserved Phases

Assume an event is enabled for 10 seconds, but actually scheduled by hardware counters for 2.5 seconds, with an original count of 200. Scaling by time ratio yields 200 × 10 / 2.5 = 800. This implies the assumption that the sampled period represents the unsampled period; if it happens to cover only the initialization peak, while the later large amount of time is in another work phase, 800 may severely deviate from the true total.

Related ratios are best derived from the same event group and compatible scope, but event groups are not a free guarantee: if hardware resources are insufficient, the entire group may fail to run. When encountering not supported, first check the event and PMU; when encountering not counted, check scheduling resources and actual running time. Do not treat these statuses as numerical zero for division. perf stat scaling and grouping

Why can time scaling still be wrong?

The full bars are ground truth supplied by the model; real measurements lack unseen intervals. Observing the first two seconds scales 200 to 1000 instead of 280; a late low-rate window underestimates. With running=0, report no valid count, not zero events.

$$\hat{N}=N_{raw}\frac{t_{enabled}}{t_{running}},t_{running}>0$$

Sampling: Locate Execution Position When Events Occur

perf record -o perf.data -F 99 -e cycles:u --call-graph dwarf -- ./prog
perf report -i perf.data --stdio
perf annotate -i perf.data --symbol=my_function

This set of commands selects the user-space cycle event, samples at a target frequency of 99 Hz, and collects user stack information. The actual sampling frequency may be throttled by the kernel or affected by the workload; do not rely on default events or frequencies of a specific version.

-F refers to the target sampling frequency, while -c refers to the event sampling period; their meanings are different. Cycles samples answer the distribution of events when the CPU is active; during blocking periods, threads are not running, and standard on-CPU stacks will not directly tell you the complete wait time. To interpret lock waits or I/O latency, combine with scheduling events or entry/return tracing.

There are trade-offs in call stack methods: the frame pointer method usually requires the relevant code to preserve frame pointers; the DWARF method records the user stack for unwinding, requiring corresponding unwind information, and the sampled stack size affects data volume and truncation; LBR depends on hardware support. -g is not a switch that guarantees a complete call chain for any binary. perf record

Symbol names, source code lines, and call chains are different pieces of information. Preserve the matched built binary, build-id, and debug symbols; avoid using files from another compilation to interpret an old perf.data. Inlining, tail calls, and compiler optimizations affect source correspondence; Linking and Loading explains the relationship between these artifacts.

Flame Graphs: Aggregating Call Stacks

In standard aggregated flame graphs, the left-right position has no temporal meaning; width represents the amount or weight of samples assigned to that call stack. Colors usually just distinguish boxes and do not inherently represent temperature or time consumption levels. Looking up along the wide frame finds deep hotspots, but hotspots may also be caused by upper layers calling repeatedly. Brendan Gregg: Flame Graphs

When the FlameGraph scripts are in PATH:

perf script -i perf.data > out.perf
stackcollapse-perf.pl out.perf > out.folded
flamegraph.pl out.folded > flamegraph.svg

These scripts belong to the FlameGraph project and are not built-in subcommands of perf. When samples are lost, symbols are missing, or stack unwinding is broken, first fix the input quality; do not just look at whether the generated graph looks pretty. Distinguish between counting sample ratios and event-period-weighted proportions according to report settings.

self and children are two types of attribution; do not add them repeatedly

A sample stack is main → handle_request → parse. Attributed by self, it falls at the current execution position of parse; attributed by children or call chain accumulation, handle_request and main will also include this sample. Thus, the percentages of parent and child rows may add up to more than 100%, which does not mean perf calculated incorrectly.

You can view reports with and without children accumulation separately and record the percentage basis used. A very wide parent frame may execute almost no code itself, serving only as a common caller for all hotspots; in this case, changing one line at the parent function entry will not naturally eliminate child call costs. perf report attribution

When sampling is too low, first increase effective observation

At 99 Hz, a single-threaded CPU-intensive phase running for only about 20 ms has an ideal magnitude of only about two samples, and actual results are further affected by events and scheduling. At this point, "a function accounts for 50%" may correspond to only one sample, and it is not worth ranking precisely based on this.

You can have the same workload repeat enough times within one process, handling initialization and steady state separately; do not change the input scale to a completely different algorithm path just to increase samples. Increasing frequency also has costs, potentially expanding sample data, losing events, or changing the observed program. First verify the number of effective samples, loss prompts, and call stack quality before considering increasing the frequency.

Does a wide parent mean expensive self time?

By default handle occupies 90% but has zero self weight: all those samples end in parse or send. main’s 100%, handle’s 90% and parse’s 60% overlap, so adding them does not give total time. Width changes track weights, not execution order.

Precise Sampling: Reduce Position Skid

Ordinary sampling may have skid between event overflow and recording the instruction position; the instruction marked as a hotspot may not be the instruction that triggered the event. Precise sampling capabilities depend on specific PMUs and event bindings; Intel PEBS and AMD IBS are different hardware mechanisms.

The p modifier in perf indicates the requirement for sampling IP precision; pp requests zero skid, and ppp's strict support is determined by the event and hardware. Do not write cycles:ppp as the "highest precision mode" applicable to all machines. First query supported precise events, then interpret results based on their event definitions. perf list event modifiers

Precise IP does not mean it has been proven that the instruction is the root cause: cache hierarchy, data sources, dependency chains, or upstream control flow still require further observation. Top-down metrics can classify execution slots and other resources by microarchitecture, but support scope and metric definitions should be confirmed according to the local PMU; do not fix them as "the same after a certain generation."

Dynamic Probing and Observation Scope

perf probe -x ./prog --add 'work=my_function'
perf probe --list

The tool will report the actual event name; then pass that name to perf record -e. Command parsing of symbols requires that the target indeed has probeable locations; source code line and local variable parsing may also require DWARF. Inlining, optimized-out variables, or one symbol corresponding to multiple locations cannot be simplified to "any function, any line can be probed."

Recording parameters must verify the target function ABI and probe location; do not directly apply x86-64 register examples to arm64. At the end, delete the probe created this time using the full actual event name, rather than clearing the shared tracing facility. perf probe manual

Attaching to a process uses -p PID, and threads use -t TID. System-wide short collections can use perf record -a ... -- sleep 10, but this observes other workloads; when selecting CPU, process, user-space, or kernel-space, ensure the conclusion matches the actual scope.

Constrain Expected Benefits with Request Time Accounting

Assume CPU samples are time-representative; parse accounts for 40% of 20 ms CPU work, which is 8 ms; accelerating it by two times only saves 4 ms. If the request also has 100 ms of serial wait, the total drops from 120 ms to 116 ms, not "hotspot accelerated by two times, interface also becomes twice as fast."

from math import isclose
cpu_ms, wait_ms = 20, 100
hot_ms = cpu_ms * 0.4
before = cpu_ms + wait_ms
after = before - hot_ms + hot_ms / 2
assert (before, after) == (120, 116)
assert isclose(1 - after / before, 1 / 30)

Multi-threaded pipelines, asynchronous overlap, and queuing systems cannot simply add the CPU time of each thread to the single-request latency. Under high load, reducing CPU service time may also reduce queuing; under low load, it may show almost no latency change. Record load level, completed throughput, and latency distribution simultaneously to avoid extrapolating the single-request model to all scenarios.

Form Verifiable Optimization Conclusions

First fix the input and completed work, then count, then sample to locate candidates; propose an explainable mechanism and modify it, then return to the same end-to-end metric for verification. Amdahl's Law reminds us that even if a part accounting for 10% of total time is completely eliminated, it can only reduce the original time by 10%.

If the hotspot proportion drops but request time does not change, the bottleneck may have shifted, the load may have changed, or the original metric may only cover part of the CPU. Preserve commands, machine information, symbol versions, and baselines so that "looks faster" becomes a reproducible conclusion.

Keywords: perf, perf_events, PMU, IPC, multiplexing, call graph, flame graph, PEBS, IBS, skid, sampling

Retain the Minimum Evidence Package for Each Conclusion

Preserve source code or build identifiers, binary build-id, complete commands, event modifiers, input scale, completed quantity, number of runs, and measurement environment. For sampling, additionally record the call stack method, data file, and loss/throttling status; for counting, additionally record event running ratio and unsupported items.

If the new version is only 1% faster than the baseline, and the fluctuation of repeated runs is larger than this, first improve the experiment or add representative repetitions; do not rush to attribute small differences to changes. You can alternate A/B execution to reduce interference from time drift, and report cold start and steady state separately. The final conclusion should land on "reduced time or resources by X under given load"; perf metrics are used to explain mechanisms.