How to profile my C++ application on linux

Question

I would like to profile my c++ application on linux. I would like to find out how much time my application spent on CPU processing vs time spent on block by IO/being idle.

I know there is a profile tool call valgrind on linux. But it breaks down time spent on each method, and it does not give me an overall picture of how much time spent on CPU processing vs idle? Or is there a way to do that with valgrind.

Answer 1

google-perf-tools - much faster alternative to callgrind (and it can generate output with the same format as callgrind, so you can use KCacheGrind).

Answer 2

If your app simply runs "flat out" (ie it's either using CPU or waiting for I/O) until it exits, and there aren't other processes competing, just do time myapp (or maybe /usr/bin/time myapp, which produces slightly different output to the shell builtin).

This will get you something like:

real    0m1.412s
user    0m1.288s
sys     0m0.056s

In this case, user+sys (kernel) time account for almost all the real time and there's just 0.068s unaccounted for... (probably time spent initally loading the app and its supporting libs).

However, if you were to see:

real    0m5.732s
user    0m1.144s
sys     0m0.078s

then your app spent 4.51s not consuming CPU and presumably blocked on IO. Which is the information I think you're looking for.

However, where this simple analysis technique breaks down is:

• Apps which wait on a timer/clock or other external stimulus (e.g event-driven GUI apps). It can't distinguish time waiting on the clock and time waiting on disk/network.
• Multithreaded apps, which need a bit more thinking about to interpret the numbers.

Answer 3

See this post.

And this post.

Basically, between the time the program starts and when it finishes, it has a call stack. During I/O, the stack terminates in a system call. During computation, it terminates in a typical instruction.

Either way, if you can sample the stack at random wall-clock times, you can see exactly why it's spending that time.

The only remaining point is - thousands of samples might give a sense of confidence, but they won't tell you much more than 10 or 20 samples will.