I'm wrapping a C function in python where the function allows a callback into python. I have the wrapper working, but with a reference counting problem that I'd like help to understand and fix. [There are few working examples of language callbacks using SWIG that I could find so I could easily have got something quite wrong.]
A simple C function, func(), that requires a callback is defined in cb_test.i. The func() creates a C array and passes that array as a param to the callback. The func() is wrapped to allow both the function to be called from python, and also for the callback to be defined in python. A simple client executing this is in test.py. The program works like this [I've cut the example down to a minimum but it's still not 'short']:
- func() is called from python, with a python callback, my_cb().
- a %pythoncode wrapper for func() creates an %array_class array, py_vals[], which is used later to optimise data marshalling between the C and python callbacks.
- the SWIG wrapper has a typemap for the callback_func typedef which maps it onto cb_func(), which is defined in cb_test.i. The python callback is stored in the py_cb_func global, and invoked by cb_func().
- the C func() is a simple function which seeds a local array with square values, allows the callback to modify them, and prints the new values.
- cb_func() is mainly a trampoline which executes my_cb() in python. The array param has to be copied from C to python data, and again in reverse after the callback has executed. We use py_vals[] for this purpose since it is only allocated once.
- my_cb() is executed in python.
My main problem is an apparent reference count problem in the %pythoncode wrapper. I have found it necessary to point a temp variable at py_vals[] to avoid python crashing somewhere in _wrap_func(). If the cb_test.use_tmp_hack variable is set to False, python will crash. Can anyone point out why python crashes, and how the code can be changed 'properly'. Since the problem is with a python variable that doesn't appear to go out of scope, I am a bit confused.
My second question is, is there a better way of marshalling the array between the C and python callbacks? I would prefer to just copy pointers to data between them rather than copying the array. But, the C func() creates and owns the C array and that cannot be changed.
Many thanks
test.py
"""Test for SWIG callback reference count problem
"""
import cb_test
# If False, python crashes from a reference count problem
cb_test.use_tmp_hack = True
""" Callback for func.
Multiplies a small array (seeded to square values) by a constant factor.
The C func() prints the modified array.
"""
def my_cb(vals, py_cb_data):
factor = py_cb_data
for i in range(cb_test.dims):
vals[i] = vals[i] * factor
return True;
# func uses my_cb as the callback function, with 4 as the callback data
cb_test.func(my_cb, 4)
cb_test.i
%module cb_test
/* Wrap callback_func param with cb_func, from where the python callback will be executed */
%typemap(in) callback_func {
py_cb_func = $input;
$1 = cb_func;
}
#ifdef SOLUTION
/* This block solves the reference counting problem */
/* The PyObject * fields in cb_data_struct require their python refcnts managed. */
%typemap(memberin) PyObject * {
Py_DecRef($1);
$1 = $input;
Py_IncRef($1);
}
#endif
%inline %{
#include "stdio.h"
#define dims 6
/* Define a C function and callback. We're wrapping both in Python. */
typedef void (*callback_func) (short Vals[], void *cb_args);
/* Func seeds a small array with square values, which the callback modifies.
The modified array is printed to stdout */
void func(callback_func cb, void *cb_args)
{
int i;
short vals[dims];
for (i = 0; i < dims; i++)
vals[i] = i * i;
cb(vals, cb_args);
for (i = 0; i < dims; i++) printf("%d ", vals[i]); printf("\n");
}
/* The cb_data struct for callback_func */
typedef struct cb_data_struct {
PyObject *py_vals;
PyObject *py_cb_data;
} cb_data_struct;
%}
%{
static PyObject *py_cb_func;
static void cb_func(short Vals[], cb_data_struct *cb_data)
{
PyObject *py_cb_data;
short *py_vals;
unsigned int i;
/* py_vals must be a ushortArray object */
SWIG_ConvertPtr(cb_data->py_vals, &py_vals, SWIGTYPE_p_ushortArray, 0);
/* Copy the C input array to the py_vals Python wrapping object */
for (i = 0; i < dims; i++)
py_vals[i] = Vals[i];
/* Pass python callback data back to python unmodified */
py_cb_data = cb_data->py_cb_data;
/* The Python callback will modify the array */
PyObject_CallFunctionObjArgs(py_cb_func, cb_data->py_vals, py_cb_data, NULL);
/* Copy the modified py_vals back to the C array */
for (i = 0; i < dims; i++)
Vals[i] = py_vals[i];
}
%}
/* Define a C array type for use in python code */
%include carrays.i
%array_class(unsigned short, ushortArray);
/* Marshal the python objects enabling the C callback_func to execute the Python callback function */
%pythoncode %{
use_tmp_hack = True
def func(cb_func, py_cb_data):
cb_data = cb_data_struct()
# If use_tmp_hack is set to False by the client, python crashes; the py_vals
# array disappears in the setters for py_vals.
# Using tmp_vals to bump the reference count seems to work around that.
if use_tmp_hack:
tmp_vals = ushortArray(dims)
cb_data.py_vals = tmp_vals
else:
cb_data.py_vals = ushortArray(dims)
cb_data.py_cb_data = py_cb_data
_cb_test.func(cb_func, cb_data)
%}
