How do I debug an MPI program?

Question

I have an MPI program which compiles and runs, but I would like to step through it to make sure nothing bizarre is happening. Ideally, I would like a simple way to attach GDB to any particular process, but I'm not really sure whether that's possible or how to do it. An alternative would be having each process write debug output to a separate log file, but this doesn't really give the same freedom as a debugger.

Are there better approaches? How do you debug MPI programs?

Answer 1

Many of the posts here are about GDB, but don't mention how to attach to a process from startup. Obviously, you can attach to all processes:

mpiexec -n X gdb ./a.out

But that is wildly ineffective since you'll have to bounce around to start up all of your processes. If you just want to debug one (or a small number of) MPI process, you can add that as a separate executable on the command line using the : operator:

mpiexec -n 1 gdb ./a.out : -n X-1 ./a.out

Now only one of your processes will get GDB.

Answer 2

http://github.com/jimktrains/pgdb/tree/master is a utility I wrote to do this very thing. There are some docs and feel free to pm me for questions.

You basically call a perl program that wraps GDB and funnels it's IO to a central server. This allows GDB to be running on each host and for you to access it on each host at the terminal.

Answer 3

The command to attach gdb to an mpi process is incomplete, it should be

mpirun -np <NP> xterm -e gdb ./program 

A brief discussion of mpi and gdb can be found here

Answer 4

I do some MPI-related debugging with log traces, but you can also run gdb if you're using mpich2: MPICH2 and gdb. This technique is a good practice in general when you're dealing with a process that's tricky to launch from a debugger.

Answer 5

As someone else said, TotalView is the standard for this. But it will cost you an arm and a leg.

The OpenMPI site has a great FAQ on MPI debugging. Item #6 in the FAQ describes how to attach GDB to MPI processes. Read the whole thing, there are some great tips.

If you find that you have far too many processes to keep track of, though, check out Stack Trace Analysis Tool (STAT). We use this at Livermore to collect stack traces from potentially hundreds of thousands of running processes and to represent them intelligently to users. It's not a full-featured debugger (a full-featured debugger would never scale to 208k cores), but it will tell you which groups of processes are doing the same thing. You can then step through a representative from each group in a standard debugger.

Answer 6

There is also my open-source tool, padb, which aims to help with parallel programming. I call it a "Job Inspection Tool" as it functions not only as a debugger can also function for example as a parallel top like program. Run in "Full Report" mode it'll show you stack traces of every process within your application along with local variables for every function over every rank (assuming you compiled with -g). It'll also show you the "MPI message queues", that is the list of outstanding sends and receives for each rank within the job.

As well as showing the full report it's also possible to tell padb to zoom in on individual bits of information within the job, there are a myriad of options and configuration items to control what information is shown, see the web page for more details.

Padb