Python CFFI memory management issues

Question

I am programming on Ubuntu, with Python 2.7.3.

I am using CFFI to populate a Python list with values coming from some C code.
That list is quite big : around 71 000 characters long when printed.

The C code is using many libraries. Hence, the following code is only here for a better understanding of what is happening.

datas_list = []
for i in range( 0, x ):
    c_pDataStructure = ffi.new( "c_DataStructure[]", 1 )    // Create a pointer to the data structure
    c.SomeCFunction( c_pDataStructure )    // Populate the data structure
    datas_list.append( c.GetSomeInfo( c_pDataStructure ) )    // Get some info from the data structure
    c.FreeDataStructure( c_pDataStructure )    // Release dynamically allocated memory

The program runs well using Wingware IDE but ends with a glibc error (*** glibc detected *** python: free(): invalid next size (fast): 0x0000000003b0b080 ***) when started from the command line, right before:

c_pDataStructure = ffi.new( "c_Datastructure[]", 1)  

After reading wim's answer, I checked if both the IDE and the command line were running the code using the same interpreter — they are (/usr/bin/python).

EDIT (valgrind report):

==5089== Process terminating with default action of signal 11 (SIGSEGV)  
==5089==  General Protection Fault  
==5089==    at 0x54FBB0: PyObject_Malloc (in /usr/bin/python2.7)  
==5089==    by 0x10B30625: allocate_owning_object (_cffi_backend.c:2972)  
==5089==    by 0x10B40EE8: allocate_with_allocator.constprop.84 (_cffi_backend.c:3032)  
==5089==    by 0x10B41010: direct_newp (_cffi_backend.c:3153)  
==5089==    by 0x10B4138C: b_newp (_cffi_backend.c:3177)  
==5089==    by 0x4F95A4: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x5008C1: PyEval_EvalCodeEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9AB7: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)

EDIT:
Here are some more information about the C data structure. This is how it looks :

typedef struct _STRUCT3{
    some int, char*
}STRUCT3, *PSTRUCT3;

typedef struct _STRUCT2{
    some int
    PSTRUCT3 pStruct3;
}STRUCT3, *PSTRUCT3;

typedef struct _STRUCT1{
    some int, char*
    PSTRUCT2 pStruct2;
}STRUCT1, *PSTRUCT1;

I made a little C program to allocate / deallocate a full C structure and valgrind did not find any memory leak.

Questions:

• What does the above valgrind report exactly mean?
• What could be the differences between running the program from the IDE and from the command line?
  Note: The IDE uses the Python argument -u (unbuffered) to run the program, but adding it to the command line makes no difference.
• As I deallocate the structure on my own, is Python's garbage collector acting? Should I use ffi.gc( c_pDataStructure, c.FreeDataStructure ) instead?

Answer 1

I found how to fix my issues:

I used ffi.gc(cdata, destructor) to create the structure. My Python code now looks like:

data_list = []
for i in range( 0, x ):  
    # Create a pointer to the data structure and tell the garbage collector how to destroy it
    gc_c_pDataStructure = ffi.gc( c.CreateDataStructure(), c.FreeDataStructure )
    c.SomeCFunction( gc_c_pDataStructure )  # Populate the data structure
    datas_list.append( c.GetSomeInfo( gc_c_pDataStructure ) ) # Store some data  

Here are some links related to ffi.gc():

• Python CFFI documentation
• David's thread

And here is the C function to create the data structure (according to the structure example from the question):

 PSTRUCT1 CreateDataStructure()
 {
     PSTRUCT1 pStruct1 = ( PSTRUCT1 ) malloc( sizeof( STRUCT1 ) );
     _SetDummyValues( pStruct1 );

     return pStruct1;
 }

As you can see, I had to create the function void _SetDummyValues( PSTRUCT1 pStruct1 ). That function sets the given structure pointers to NULL.

Answer 2

This may have also been related to this bug which Coverity found in some of our CFFI-generated code:

  x0 = (void *)alloca((size_t)datasize);
  ...
  { 
      free(x0); 
  }

As you can see, it's calling free on stack-allocated memory.