I've got a question regarding passing of a std::vector of structs via MPI.
First off, details. I'm using OpenMPI 1.4.3 (MPI-2 compliant) with gcc.
Note that I can't use boost MPI or OOMPI -- I'm bound to using this version.
I've got a struct to aggregate some data:
struct Delta {
Delta() : dX(0.0), dY(0.0), dZ(0.0) {};
Delta(double dx, double dy, double dz) :
dX(dx), dY(dy), dZ(dz) {};
Delta(const Delta& rhs) :
dX(rhs.dX), dY(rhs.dY), dZ(rhs.dZ) {};
double dX;
double dY;
double dZ;
};
typedef std::vector<Delta> DeltaLine;
and I have a DeltaLine that I'd like to broadcast, via MPI, to all the nodes.
Can I do the following safely and portably?
This works for me in my test case. I just want to make sure it's legal and kosher across different platforms and according to the C++ and MPI standards.
Thanks!
Madeleine.
//Create an MPI struct for the Delta class
const int nItems=3;
int blocklengths[nItems] = {1, 1, 1};
MPI_Datatype types[nItems] = {MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE};
MPI_Datatype MPI_DeltaType;
MPI_Aint offsets[nItems];
offsets[0] = offsetof(Delta, dX);
offsets[1] = offsetof(Delta, dY);
offsets[2] = offsetof(Delta, dZ);
MPI_Type_create_struct(nItems, blocklengths, offsets, types, &MPI_DeltaType);
MPI_Type_commit(&MPI_DeltaType);
//This is the vector to be filled, and its size
DeltaLine deltaLine;
unsigned deltaLineSize;
//If this is the master proc, get the DeltaLine and its size
if(amMaster()) {
deltaLine = getMasterDeltaLine();
deltaLineSize = deltaLine.size();
}
//Send out the correct size
MPI_Bcast(&deltaLineSize, 1, MPI_UNSIGNED, COMM_PROC, MPI_COMM_WORLD);
//Size the delta line vector, and broadcast its contents
deltaLine.reserve(deltaLineSize);
MPI_Bcast(&deltaLine.front(), deltaLineSize, MPI_DeltaType, COMM_PROC, MPI_COMM_WORLD);
//Free up the type
MPI_Type_free(&MPI_DeltaType);
The C++ standard guarantees that the elements of std::vector
are stored contiguously in memory and that std::vector::reserve()
(re-)allocates memory if necessary at the time of call, therefore your solution is perfectly valid from memory management point of view. Though, as Solkar noted, std::vector::reserve()
only reserves memory space but the vector object is not aware that there is data being directly written in that memory and therefore keeps the previous element count (zero for freshly created vectors). This can be fixed by calling std::vector::resize()
before the second broadcast operation.
One comment though that applies to all cases when constructed MPI datatypes are used to send arrays - you should take care of possible padding between the consecutive array elements. In other words, it is possible for the following to hold because of possible padding at the end of the struct
:
(char*)&deltaLine[1] - (char*)&deltaLine[0] != mpi_extentof(MPI_DeltaType)
where mpi_extentof
is the extent of the MPI datatype as returned by MPI_Type_get_extent()
. Because MPI uses the extent to determine where each array element starts, it is advisable to explicitly set it for any structure type that is used to send more than one element. With MPI-1 this is typically done by adding one special structure element of the MPI_UB
pseudotype, but in modern MPI code (or in MPI-2 in general) one should use MPI_Type_create_resized
for that purpose:
//Create an MPI struct for the Delta class
const int nItems=3;
int blocklengths[nItems] = {1, 1, 1};
MPI_Datatype types[nItems] = {MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE};
MPI_Datatype MPI_DeltaType_proto, MPI_DeltaType;
MPI_Aint offsets[nItems];
offsets[0] = offsetof(Delta, dX);
offsets[1] = offsetof(Delta, dY);
offsets[2] = offsetof(Delta, dZ);
MPI_Type_create_struct(nItems, blocklengths, offsets, types, &MPI_DeltaType_proto);
// Resize the type so that its length matches the actual structure length
// Get the constructed type lower bound and extent
MPI_Aint lb, extent;
MPI_Type_get_extent(MPI_DeltaType_proto, &lb, &extent);
// Get the actual distance between to vector elements
// (this might not be the best way to do it - if so, substitute a better one)
extent = (char*)&deltaLine[1] - (char*)&deltaLine[0];
// Create a resized type whose extent matches the actual distance
MPI_Type_create_resized(MPI_DeltaType_proto, lb, extent, &MPI_DeltaType);
MPI_Type_commit(&MPI_DeltaType);
In your case there are only double
elements in the structure and no padding is expected, therefore doing this all is not necessary. But keep it in mind for your future work with MPI.
std::vector::reserve(N)
does not affect size
but (if at all) capacity
(and maybe location), so for the receiving containers deltaLine
will still be a zero size vector, regardless of its capacity
equaling deltaLineSize
.
That's not yet a problem in the code as-is, but I assume you intend to do some processing using the received data.
I would also check the return value of (at least) the first MPI_BCast
, because if that, for whatever reason fails on one process, the vector's size there would be 0, and if it responds to the second broadcast bounds will be violated.