How should strace be used?

Question

A colleague once told me that the last option when everything has failed to debug on Linux was to use strace.

I tried to learn the science behind this strange tool, but I am not a system admin guru and I didn’t really get results.

So,

• What is it exactly and what does it do?
• How and in which cases should it be used?
• How should the output be understood and processed?

In brief, in simple words, how does this stuff work?

Answer 1

strace is a good tool for learning how your program makes various system calls (requests to the kernel) and also reports the ones that have failed along with the error value associated with that failure. Not all failures are bugs. For example, a code that is trying to search for a file may get a ENOENT (No such file or directory) error but that may be an acceptable scenario in the logic of the code.

One good use case of using strace is to debug race conditions during temporary file creation. For example a program that may be creating files by appending the process ID (PID) to some predecided string may face problems in multi-threaded scenarios. [A PID+TID (process id + thread id) or a better system call such as mkstemp will fix this].

It is also good for debugging crashes. You may find this (my) article on strace and debugging crashes useful.

Answer 2

Strace stands out as a tool for investigating production systems where you can't afford to run these programs under a debugger. In particular, we have used strace in the following two situations:

• Program foo seems to be in deadlock and has become unresponsive. This could be a target for gdb; however, we haven't always had the source code or sometimes were dealing with scripted languages that weren't straight-forward to run under a debugger. In this case, you run strace on an already running program and you will get the list of system calls being made. This is particularly useful if you are investigating a client/server application or an application that interacts with a database
• Investigating why a program is slow. In particular, we had just moved to a new distributed file system and the new throughput of the system was very slow. You can specify strace with the '-T' option which will tell you how much time was spent in each system call. This helped to determine why the file system was causing things to slow down.

For an example of analyzing using strace see my answer to this question.

Answer 3

strace lists all system calls done by the process it's applied to. If you don't know what system calls mean, you won't be able to get much mileage from it.

Nevertheless, if your problem involves files or paths or environment values, running strace on the problematic program and redirecting the output to a file and then grepping that file for your path/file/env string may help you see what your program is actually attempting to do, as distinct from what you expected it to.

Answer 4

strace -tfp PID will monitor the PID process's system calls, thus we can debug/monitor our process/program status.

Answer 5

Strace is a tool that tells you how your application interacts with your operating system.

It does this by telling you what OS system calls your application uses and with what parameters it calls them.

So for instance you see what files your program tries to open, and weather the call succeeds.

You can debug all sorts of problems with this tool. For instance if application says that it cannot find library that you know you have installed you strace would tell you where the application is looking for that file.

And that is just a tip of the iceberg.

Answer 6

I liked some of the answers where it reads strace checks how you interacts with your operating system.

This is exactly what we can see. The system calls. If you compare strace and ltrace the difference is more obvious.

$>strace -c cd
Desktop  Documents  Downloads  examples.desktop  Music  Pictures  Public  Templates  Videos
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0         7           read
  0.00    0.000000           0         1           write
  0.00    0.000000           0        11           close
  0.00    0.000000           0        10           fstat
  0.00    0.000000           0        17           mmap
  0.00    0.000000           0        12           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         2           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0         8         8 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         2           getdents
  0.00    0.000000           0         2         2 statfs
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         9           openat
  0.00    0.000000           0         1           set_robust_list
  0.00    0.000000           0         1           prlimit64
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                    93        10 total

On the other hand there is ltrace that traces functions.

$>ltrace -c cd
Desktop  Documents  Downloads  examples.desktop  Music  Pictures  Public  Templates  Videos
% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 15.52    0.004946         329        15 memcpy
 13.34    0.004249          94        45 __ctype_get_mb_cur_max
 12.87    0.004099        2049         2 fclose
 12.12    0.003861          83        46 strlen
 10.96    0.003491         109        32 __errno_location
 10.37    0.003303         117        28 readdir
  8.41    0.002679         133        20 strcoll
  5.62    0.001791         111        16 __overflow
  3.24    0.001032         114         9 fwrite_unlocked
  1.26    0.000400         100         4 __freading
  1.17    0.000372          41         9 getenv
  0.70    0.000222         111         2 fflush
  0.67    0.000214         107         2 __fpending
  0.64    0.000203         101         2 fileno
  0.62    0.000196         196         1 closedir
  0.43    0.000138         138         1 setlocale
  0.36    0.000114         114         1 _setjmp
  0.31    0.000098          98         1 realloc
  0.25    0.000080          80         1 bindtextdomain
  0.21    0.000068          68         1 opendir
  0.19    0.000062          62         1 strrchr
  0.18    0.000056          56         1 isatty
  0.16    0.000051          51         1 ioctl
  0.15    0.000047          47         1 getopt_long
  0.14    0.000045          45         1 textdomain
  0.13    0.000042          42         1 __cxa_atexit
------ ----------- ----------- --------- --------------------
100.00    0.031859                   244 total

Although I checked the manuals several time, I haven't found the origin of the name strace but it is likely system-call trace, since this is obvious.

There are three bigger notes to say about strace.

Note 1: Both these functions strace and ltrace are using the system call ptrace. So ptrace system call is effectively how strace works.

The ptrace() system call provides a means by which one process (the "tracer") may observe and control the execution of another process (the "tracee"), and examine and change the tracee's memory and registers. It is primarily used to implement breakpoint debugging and system call tracing.

Note 2: There are different parameters you can use with strace, since strace can be very verbose. I like to experiment with -c which is like a summary of things. Based on -c you can select one system-call like -e trace=open where you will see only that call. This can be interesting if you are examining what files will be opened during the command you are tracing. And of course, you can use the grep for the same purpose but note you need to redirect like this 2>&1 | grep etc to understand that config files are referenced when the command was issued.

Note 3: I find this very important note. You are not limited to a specific architecture. strace will blow you mind, since it can trace over binaries of different architectures.