Get gprof to profile based on wall-clock time?

Question

My understanding is that by default gprof takes into account CPU time. Is there a way to get it to profile based on wall-clock time?

My program does a lot of disk i/o, so the CPU time it uses only represents a fraction of the actual execution time. I need to know which portions of the disk i/o take up the most time.

Answer 1

You can measure wall-clock time by using profiler from google-perftools. To switch google profiler to wall-clock mode, set the environment variable CPUPROFILE_REALTIME=1.

Answer 2

You can use strace or cachegrind to profile the code properly. strace will give you details of time spent in system calls and cachegrind will give detailed analysis of resource utilization.

Answer 3

gprof won't do this. Look at this.

And this.

In a nutshell: Under gdb, get it running and do Ctrl-Break or Ctrl-C 10 times at random, and display the call stack. If your I/O is taking (for example) 60% of the time, then on (roughly) 6 out of 10 pauses, you will see it in the writebuf or readbuf routine, and the lines of code requesting that I/O will be clearly displayed on the stack.

You could also use lsstack to get the same information.

Answer 4

It is very easy to change gprof to do wall-clock profiling. The only 8 chars to replace are:

ITIMER_PROF -> ITIMER_REAL

SIGPROF -> SIGALRM

in the file glibc/sysdeps/posix/profil.c, function __profil, near the calls to setitimer and sigaction (more exact __Setitimer and __sigaction)

After the change any program which uses SIGALRM will be broken and any program which have no blocking-syscall restarting code can give wrong results.

Also, you can directly change int values in glibc binary (please, dont do this on system wide libc.so, make a separate copy and give it to the program with LD_LIBRARY_PATH)

For binary patch, ITIMER_PROF is 2; ITIMER_REAL is 0; SIGPROF is 27 (0x1b); SIGALRM is 14 (0x0e). There are two places for each constant in function profil of glibc.

Another way is to write a ptrace-debugger, which will change arguments of setitimer and sigaction functions at run-time.