MPI_reduce() with custom Datatype containing dynam

Question

I don't get why MPI_Reduce() does a segmentation fault as soon as I use a custom MPI datatype which contains dynamically allocated arrays. Does anyone know ? The following code crashes with 2 processors, inside the MPI_Reduce(). However If I remove the member double *d int MyType and changes the operator and MPI type routines accordingly, the reduction is done without any problem.

Is there a problem using dynamically allocated arrays or is there something fundamentally wrong with what I do :

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>



typedef struct mytype_s
{
    int c[2];
    double a;
    double b;
    double *d;
} MyType;



void CreateMyTypeMPI(MyType *mt, MPI_Datatype *MyTypeMPI)
{
    int block_lengths[4];                        // # of elt. in each block
    MPI_Aint displacements[4];                   // displac.
    MPI_Datatype typelist[4];                    // list of types
    MPI_Aint start_address, address;            // use for calculating displac.
    MPI_Datatype myType;

    block_lengths[0] = 2;
    block_lengths[1] = 1;
    block_lengths[2] = 1;
    block_lengths[3] = 10;

    typelist[0] = MPI_INT;
    typelist[1] = MPI_DOUBLE;
    typelist[2] = MPI_DOUBLE;
    typelist[3] = MPI_DOUBLE;

    displacements[0] = 0;

    MPI_Address(&mt->c, &start_address);
    MPI_Address(&mt->a, &address);
    displacements[1] = address - start_address;

    MPI_Address(&mt->b,&address);
    displacements[2] = address-start_address;

    MPI_Address(&mt->d, &address);
    displacements[3] = address-start_address;

    MPI_Type_struct(4,block_lengths, displacements,typelist,MyTypeMPI);
    MPI_Type_commit(MyTypeMPI);
}




void MyTypeOp(MyType *in, MyType *out, int *len, MPI_Datatype *typeptr)
{
    int i;
    int j;

    for (i=0; i < *len; i++)
    {
        out[i].a += in[i].a;
        out[i].b += in[i].b;
        out[i].c[0] += in[i].c[0];
        out[i].c[1] += in[i].c[1];

        for (j=0; j<10; j++)
        {
            out[i].d[j] += in[i].d[j];
        }
    }
}




int main(int argc, char **argv)
{
    MyType mt;
    MyType mt2;
    MPI_Datatype MyTypeMPI;
    MPI_Op MyOp;
    int rank;
    int i;

    MPI_Init(&argc,&argv);
    MPI_Comm_rank(MPI_COMM_WORLD,&rank);


    mt.a = 2;
    mt.b = 4;
    mt.c[0] = 6;
    mt.c[1] = 8;
    mt.d = calloc(10,sizeof *mt.d);
    for (i=0; i<10; i++) mt.d[i] = 2.1;

    mt2.a = 0;
    mt2.b = 0;
    mt2.c[0] = mt2.c[1] = 0;
    mt2.d = calloc(10,sizeof *mt2.d);


    CreateMyTypeMPI(&mt, &MyTypeMPI);
    MPI_Op_create((MPI_User_function *) MyTypeOp,1,&MyOp);

    if(rank==0) printf("type and operator are created now\n");

    MPI_Reduce(&mt,&mt2,1,MyTypeMPI,MyOp,0,MPI_COMM_WORLD);

    if(rank==0)
    {




        for (i=0; i<10; i++) printf("%f ",mt2.d[i]);
        printf("\n");
    }

    free(mt.d);
    free(mt2.d);
    MPI_Finalize();

    return 0;
}

Answer 1

Let's look at your struct:

typedef struct mytype_s
{
    int c[2];
    double a;
    double b;
    double *d;
} MyType;

...

MyType mt;
mt.d = calloc(10,sizeof *mt.d);

And your description of this struct as an MPI type:

displacements[0] = 0;

MPI_Address(&mt->c, &start_address);
MPI_Address(&mt->a, &address);
displacements[1] = address - start_address;

MPI_Address(&mt->b,&address);
displacements[2] = address-start_address;

MPI_Address(&mt->d, &address);
displacements[3] = address-start_address;

MPI_Type_struct(4,block_lengths, displacements,typelist,MyTypeMPI);

The problem is, this MPI struct is only ever going to apply to the one instance of the structure you've used in the definition here. You have no control at all of where calloc() decides to grab memory from; it could be anywhere in virtual memory. The next one of these type you create and instantiate, the displacement for your d array will be completely different; and even using the same struct, if you change the size of the array with realloc() of the current mt, it could end up having a different displacement.

So when you send, receive, reduce, or anything else with one of these types, the MPI library will dutifully go to a probably meaningless displacement, and try to read or write from there, and that'll likely cause a segfault.

Note that this isn't an MPI thing; in using any low-level communications library, or for that matter trying to write out/read in from disk, you'd have the same problem.

Your options include manually "marshalling" the array into a message, either with the other fields or without; or adding some predictability to where d is located such as by defining it to be an array of some defined maximum size.