SWIG reference count error with python callback

Question

I'm wrapping a C function in python where the function allows a callback into python. I have the wrapper working, but with a reference counting problem that I'd like help to understand and fix. [There are few working examples of language callbacks using SWIG that I could find so I could easily have got something quite wrong.]

A simple C function, func(), that requires a callback is defined in cb_test.i. The func() creates a C array and passes that array as a param to the callback. The func() is wrapped to allow both the function to be called from python, and also for the callback to be defined in python. A simple client executing this is in test.py. The program works like this [I've cut the example down to a minimum but it's still not 'short']:

• func() is called from python, with a python callback, my_cb().
• a %pythoncode wrapper for func() creates an %array_class array, py_vals[], which is used later to optimise data marshalling between the C and python callbacks.
• the SWIG wrapper has a typemap for the callback_func typedef which maps it onto cb_func(), which is defined in cb_test.i. The python callback is stored in the py_cb_func global, and invoked by cb_func().
• the C func() is a simple function which seeds a local array with square values, allows the callback to modify them, and prints the new values.
• cb_func() is mainly a trampoline which executes my_cb() in python. The array param has to be copied from C to python data, and again in reverse after the callback has executed. We use py_vals[] for this purpose since it is only allocated once.
• my_cb() is executed in python.

My main problem is an apparent reference count problem in the %pythoncode wrapper. I have found it necessary to point a temp variable at py_vals[] to avoid python crashing somewhere in _wrap_func(). If the cb_test.use_tmp_hack variable is set to False, python will crash. Can anyone point out why python crashes, and how the code can be changed 'properly'. Since the problem is with a python variable that doesn't appear to go out of scope, I am a bit confused.

My second question is, is there a better way of marshalling the array between the C and python callbacks? I would prefer to just copy pointers to data between them rather than copying the array. But, the C func() creates and owns the C array and that cannot be changed.

Many thanks

test.py

"""Test for SWIG callback reference count problem
"""

import cb_test

# If False, python crashes from a reference count problem
cb_test.use_tmp_hack = True

""" Callback for func.
    Multiplies a small array (seeded to square values) by a constant factor.
    The C func() prints the modified array.
"""
def my_cb(vals, py_cb_data):
    factor = py_cb_data
    for i in range(cb_test.dims):
        vals[i] = vals[i] * factor
    return True;


# func uses my_cb as the callback function, with 4 as the callback data
cb_test.func(my_cb, 4)

cb_test.i

%module cb_test

/* Wrap callback_func param with cb_func, from where the python callback will be executed */
%typemap(in) callback_func {
  py_cb_func = $input;
  $1 = cb_func;
}

#ifdef SOLUTION
/* This block solves the reference counting problem */
/* The PyObject * fields in cb_data_struct require their python refcnts managed. */
%typemap(memberin) PyObject * {
  Py_DecRef($1);
  $1 = $input;
  Py_IncRef($1);
}
#endif

%inline %{
  #include "stdio.h"

  #define dims 6

  /* Define a C function and callback. We're wrapping both in Python. */
  typedef void (*callback_func) (short Vals[], void *cb_args);

  /* Func seeds a small array with square values, which the callback modifies.
     The modified array is printed to stdout */
  void func(callback_func cb, void *cb_args)
  {
    int i;
    short vals[dims];

    for (i = 0; i < dims; i++)
      vals[i] = i * i;
    cb(vals, cb_args);
    for (i = 0; i < dims; i++)  printf("%d ", vals[i]);  printf("\n");
  }

  /* The cb_data struct for callback_func */
  typedef struct cb_data_struct {
    PyObject *py_vals;
    PyObject *py_cb_data;
  } cb_data_struct;
%}

%{
  static PyObject *py_cb_func;
  static void cb_func(short Vals[], cb_data_struct *cb_data)
  {
    PyObject *py_cb_data;
    short *py_vals;
    unsigned int i;

    /* py_vals must be a ushortArray object */
    SWIG_ConvertPtr(cb_data->py_vals, &py_vals, SWIGTYPE_p_ushortArray, 0);

    /* Copy the C input array to the py_vals Python wrapping object */
    for (i = 0; i < dims; i++)
      py_vals[i] = Vals[i];

    /* Pass python callback data back to python unmodified */
    py_cb_data = cb_data->py_cb_data;

    /* The Python callback will modify the array */
    PyObject_CallFunctionObjArgs(py_cb_func, cb_data->py_vals, py_cb_data, NULL);

    /* Copy the modified py_vals back to the C array */
    for (i = 0; i < dims; i++)
      Vals[i] = py_vals[i];
  }
%}


/* Define a C array type for use in python code */
%include carrays.i
%array_class(unsigned short, ushortArray);

/* Marshal the python objects enabling the C callback_func to execute the Python callback function */
%pythoncode %{
use_tmp_hack = True

def func(cb_func, py_cb_data):
  cb_data = cb_data_struct()

  # If use_tmp_hack is set to False by the client, python crashes; the py_vals
  # array disappears in the setters for py_vals.
  # Using tmp_vals to bump the reference count seems to work around that.
  if use_tmp_hack:
    tmp_vals = ushortArray(dims)
    cb_data.py_vals = tmp_vals
  else:
    cb_data.py_vals = ushortArray(dims)

  cb_data.py_cb_data = py_cb_data
  _cb_test.func(cb_func, cb_data)
%}

Answer 1

The crashing problem lies in the setters for the PyObject * members of cb_data_struct. I assumed that the Python interpreter would somehow manage the reference counting, but it appears necessary to do that in SWIG. I believe I've solved that using this typemap which is used in the setters for the members. It decrements the count of prior values and increments the new value.

%typemap(memberin) PyObject * {
  Py_DecRef($1);
  $1 = $input;
  Py_IncRef($1);
}

I've added this block to the code in the question, hopefully obviously.

After looking quite hard, I don't think there is a more efficient way to marshal arrays between the C and Python callbacks.

[This solution does come with a health warning because I'm not that experienced in SWIG. I have though, verified that objects get deleted when the count reduces to zero, as expected.]