perf !?

perf is a Linux utility used to measure various performance counters on Linux.

You normally find it in your distribution repositories (e.g. sudo apt install perf).

These are some quick notes on common usages, not complete nor perfect.

What is eating my CPU cycles?

sudo perf top -a 

But I want only my program!

sudo perf top -p `pidof myprogram`

I am rich and I have a lot of CPUs

sudo perf top -C <CORE_LIST>

Where <CORE_LIST> can be something line 1-8, 1,2,3, 1-2,4-5.

I don’t see all the events/the system is too slowed-down

Use -F do determine collecting frequency.

I want to see the call-graph

Or, what function chains are the more expensive?

Add -g.

I want to record and analyze later

sudo perf record <OPTIONS>

Or, if the program is not running, sudo perf record <OPTIONS> -- your command.

It will generate a perf.data file that you can analyze later:

sudo perf report

perf record eats all my disk space!

Save the results to another location:

sudo perf record -o /tmp/perf.data <OPTIONS>

Then, load them with the -i option:

sudo perf report -i /tmp/perf.data

But I see no function names!

Have you compiled with debug symbols?

Are you using gcc? Add -g to have debug symbols. You may also remove optimizations to better see the problem: -O0.

Explore the results

perf report and perf top allows you to analyze in-depth the results.

Annotate the code

You can analyze in-depth the assembly code that impact the performances: select the function with arrow keys and press a.

And the function tree?

If you have recorded with -g, pressing enter on an entry will show the functions called by that specific function.

Use --no-children to not accumulate the cost of the children in the caller functions.

Use -G to have inverted graphs: it will show who called the highlighted function instead of who is called by this.

I am lost

Press h, or consult the manual with man perf, perf report --help or look at some more extensive examples.

I hate the terminal!

You shouldn’t ;)

But if you really need, use a GUI like Hotspot, no installation required!

Firefox has a built-in analyzer. Even if not thought for general profiling, it works ;) Instructions to use it can be found here.

What about a nice visualization?

You may want to visualize your results with a flamegraph.

And the cache?

sudo perf stat

It will print statistics when killed with (e.g. with CTRL+C). Add more events with -d, restrict to a given PID with -p, to CPUs with -C, print statistics every N milliseconds with -I <N>.

You can also record and analyze similarly to perf record and perf report, but I let you learn by yourself ;).

A wide variety of events can be collected (e.g. LLC misses, LLC loads, branch predictions, …). You set what to collect with -e. A list of events is available with perf list.

I want to know everything!

A lot of examples can be found at Brendan Gregg’s site.