Trouble Understanding MPI_Type_create_struct

Question

I'm having trouble understanding the MPI_Type_create_struct method. Say we have a struct:

   struct foo(){
       float value;
       char rank;
   }

And we want to send this struct to another process. Conside the code sample below:

int count = 2; //number of elements in struct
MPI_Aint offsets[count] = {0, 8};
int blocklengths[count] = {1, 1};
MPI_Datatype types[count] = {MPI_FLOAT, MPI_CHAR};
MPI_Datatype my_mpi_type;

MPI_Type_create_struct(count, blocklengths, offsets, types, &my_mpi_type);

I'm not sure what offsets and blocklengths do in this example. Can somebody explain these two parts above?

Answer 1

The purpose of MPI_Type_create_struct() is, as you know, to provide a way to create user's MPI_Datatypes mapping his structured types. These new types will subsequently be usable for MPI communications and other calls just as the default types, allowing for example to transfer arrays of structures the same way you would transfer arrays of ints or floats.

Now let's see the function itself in more details.
Here is its synopsis as returned by the man command:

NAME
   MPI_Type_create_struct -  Create an MPI datatype from a general set of
           datatypes, displacements, and block sizes

SYNOPSIS
   int MPI_Type_create_struct(int count,
                              const int array_of_blocklengths[],
                              const MPI_Aint array_of_displacements[],
                              const MPI_Datatype array_of_types[],
                              MPI_Datatype *newtype)

INPUT PARAMETERS
   count   - number  of  blocks  (integer)  ---  also number of entries
             in arrays array_of_types, array_of_displacements and
             array_of_blocklengths
   array_of_blocklengths
           - number of elements in each block (array of integer)
   array_of_displacements
           - byte displacement of each block (array of address integer)
   array_of_types
           - type of elements in each block (array of handles to datatype
             objects)

OUTPUT PARAMETERS
   newtype - new datatype (handle)

Let's see for the input parameters if their meaning calls for further explanation:

• count: this is quite clear, and in your case, that would be 2
• array_of_types: well, that'd be { MPI_FLOAT, MPI_CHAR } for your example
• array_of_blocklengths: again, not much to say. { 1, 1 } is what you need here
• array_of_displacements: this is the one for which you have to be a bit more careful. It corresponds to the memory address offsets from the start of the structure, to the address of each element listed in array_of_types. In your case, that would be something like { &f.value - &f, &f.rank - &f }, with f being of type foo. The tricky part here is that, because of potential alignment constraints, you cannot be sure that this will be equal to { 0, sizeof( float ) } (although here I'm pretty sure it will be). Therefore, using addresses offsets as shown makes the method fully portable. Moreover (thx Hristo Iliev to pointing it to me) you can (and should) use the offsetof() macro from stddef.h which does exactly this pointer arithmetic for your, simplifying the code to { offsetof( foo, value ), offsetof( foo, rank ) } which looks nicer.

With the arguments initialised this way, the call to MPI_Type_create_struct() will return a new MPI_Datatype, which will be suitable for sending or receiving one foo at the time. The reason for that is that this new type doesn't take into account the actual extent of the structure, including the alignment constraints for its fields. And you example is perfect in this regards since it will (very likely) be hollow.

The reason for that is that floats have in general an alignment constraint of 32b, while chars have none. Therefore, the starting address of the second structure foo of an array of theme is not right at the end of the first one. It is at the next 32b-aligned memory address. This will leave us with a hole of 3 Bytes between the end of an element of the structure to the start of the next in the array.

To handle this issue, you'll have to resize your type for extending it with MPI_Type_create_resized(), which synopsis is as follow:

NAME
   MPI_Type_create_resized -  Create a datatype with a new lower bound
        and extent from an existing datatype

SYNOPSIS
   int MPI_Type_create_resized(MPI_Datatype oldtype,
                               MPI_Aint lb,
                               MPI_Aint extent,
                               MPI_Datatype *newtype)

INPUT PARAMETERS
   oldtype - input datatype (handle)
   lb      - new lower bound of datatype (address integer)
   extent  - new extent of datatype (address integer)

OUTPUT PARAMETERS
   newtype - output datatype (handle)

Using it is quite easy as both lb and extend can be retrieved by directly calling a function specifically meant for this purpose, namely MPI_Type_get_extent() (but actually, you could also directly use 0 and sizeof( foo )). In addition, since the intermediary type used for calling MPI_Type_get_extent() and MPI_Type_create_resized() isn't used in any actual MPI communication, it doesn't need to be committed with MPI_Type_commit(), sparing you some calls and time.

Now, with that, your code becomes:

int count = 2;
int array_of_blocklengths[] = { 1, 1 };
MPI_Aint array_of_displacements[] = { offsetof( foo, value ),
                                      offsetof( foo, rank ) };
MPI_Datatype array_of_types[] = { MPI_FLOAT, MPI_CHAR };
MPI_Datatype tmp_type, my_mpi_type;
MPI_Aint lb, extent;

MPI_Type_create_struct( count, array_of_blocklengths, array_of_displacements,
                        array_of_types, &tmp_type );
MPI_Type_get_extent( tmp_type, &lb, &extent );
MPI_Type_create_resized( tmp_type, lb, extent, &my_mpi_type );
MPI_Type_commit( &my_mpi_type );