When do I need to use MPI_Barrier()?

Question

I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Answer 1

All collective operations in MPI before MPI-3.0 are blocking, which means that it is safe to use all buffers passed to them after they return. In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!) So MPI_Barrier is not necessary (or very helpful) before/after collective operations, if all buffers are valid already.

Please also note, that MPI_Barrier does not magically wait for non-blocking calls. If you use a non-blocking send/recv and both processes wait at an MPI_Barrier after the send/recv pair, it is not guaranteed that the processes sent/received all data after the MPI_Barrier. Use MPI_Wait (and friends) instead. So the following piece of code contains errors:

/* ERRORNOUS CODE */

Code for Process 0:
Process 0 sends something using MPI_Isend
MPI_Barrier(MPI_COMM_WORLD);
Process 0 uses buffer passed to MPI_Isend // (!)

Code for Process 1:
Process 1 recvs something using MPI_Irecv
MPI_Barrier(MPI_COMM_WORLD);
Process 1 uses buffer passed to MPI_Irecv // (!)

Both lines that are marked with (!) are unsafe!

MPI_Barrier is only useful in a handful of cases. Most of the time you do not care whether your processes sync up. Better read about blocking and non-blocking calls!

Answer 2

One use of MPI_Barrier is for example to control access to an external resource such as the filesystem, which is not accessed using MPI. For example, if you want each process to write stuff to a file in sequence, you could do it like this:

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
for ( int ii = 0; ii < size; ++ii ) {
    if ( rank == ii ) {
        // my turn to write to the file
        writeStuffToTheFile();
    }
    MPI_Barrier(MPI_COMM_WORLD);
}

That way, you can be sure that no two processes are concurrently calling writeStuffToTheFile.

Answer 3

May MPI_Barrier() is not often used, but it is useful. In fact, even if you were use the synchronous communication, the MPI_Send/Recv() can only make sure the two processes is synchronized. In my project, a cuda+MPI project, all i used is asynchronous communication. I found that in some cases if i dont use the MPI_Barrier() followed by the Wait() function, the situation that two processes(gpu) want to transmit data to each other at the same time is very likely to happen, which could badly reduce the program efficiency. The bug above ever divers me mad and take me a few days to find it. Therefore you may think carefully whether use the MPI_Barrier() when you used the MPI_Isend/Irecv in your program. Sometimes sync the processes is not only neccessary but also MUST, especially ur program is dealing with the device.