MPI, C, derived types, struct of vectors?

Question

I need to create an MPI derived type to represent a class in my program. The class is fairly straight forward, but large (about 75 data members**). All the data members are single values, 1D arrays, or 2D arrays. Here is an example:

class RestartData {
  int dsr;
  double firea2sorgn;
  int ifwoody[NUM_PFT];
  double rootfrac[MAX_ROT_LAY][NUM_PFT];
  ....
  ....
}

I think that using the MPI_Type_struct is appropriate. (e.g. http://www.open-mpi.org/doc/v1.5/man3/MPI_Type_struct.3.php)

And I more or less follow the example in this question: struct serialization in C and transfer over MPI, but I am not sure how to handle the 2D arrays. Can I make an MPI_Type_struct that contains several MPI_Type_vectors? I have been unable to find an example of creating an MPI_Type_struct containing 2D arrays. Am I on the right approach?

Thanks in advance.

---

** I think I understand the possible problems with passing a single large message, but in this case, the message is passed infrequently, and at a natural synchronization point (slaves sending data back to the master when they are done crunching numbers)

Answer 1

Derived types in MPI can be freely constructed from other derived types and then used to further create other derived types.

2D arrays, as long as they are contiguous in memory as in your case, are not that different from 1D arrays. When it comes to the rootfrac member, you could either create a contiguous datatype with MAX_ROOT_LAY * NUM_PFT elements of type MPI_DOUBLE or you could create a contiguous datatype (let's call it t_dbl_pft) with NUM_PFT elements of type MPI_DOUBLE and then use it to create another contiguous datatype with MAX_ROOT_LAY elements of type t_dbl_pft. Another option is to not create a datatype at all since the MPI structured type constructor takes a separate block length (i.e. number of elements) for each element of the structure.

For example, the following type describes the data members that you've shown:

#include <cstddef> // for offsetof

MPI_Type t_1d;
MPI_Type_contiguous(NUM_PFT, MPI_DOUBLE, &t_1d);
// No need to commit - this type is not used for communication

MPI_Type t_restart;
int counts[4] = { 1, 1, NUM_PFT, MAX_ROT_LAY };
MPI_Type types[4] = { MPI_INT, MPI_DOUBLE, MPI_INT, t_1d };
MPI_Aint displs[4] = {
   offsetof(RestartData, dsr),
   offsetof(RestartData, firea2sorgn),
   offsetof(RestartData, ifwoody),
   offsetof(RestartData, rootfrac),
};
MPI_Type_create_struct(4, counts, displs, types, &t_restart);
MPI_Type_commit(&t_restart);

// The 1D type is no longer needed
MPI_Type_free(&t_1d);

Note that you have to either create the MPI datatype inside a member function of RestartData or declare the routine where the type is created as friend to the class so that it could access the private and protected data members. Also note that offsetof only works with POD (plain old data) classes, e.g. no fancy constructors and members of types like std::string.

Answer 2

The easiest way to go is to just treat the whole object as a big buffer:

MPI_Datatype datatype;

MPI_Type_contiguous(sizeof(RestartData), MPI_BYTE, &datatype);
MPI_Type_commit(&datatype);

I don't see the benefit of telling MPI about the internal structure of your class.

Alternatively a 2D array is an array of 1D arrays. So (I imagine) you could use one call to MPI_Type_contiguous per array dimension to build up the array datatype.