Cython has “first class function objects” - how ef

I have an external library that computes the optima, say minima, of a given function. Say its headers give me a function

double[] minimizer(ObjFun f)

where the headers define

typedef double (*ObjFun)(double x[])

I have generated Cython wrappers for this library. I now want to give user parameterized functions to it, specifically, I want to write a function

def getFunction(double q11, double q12, double q22):
    cdef f(double x[]):
        return x[0]*x[0]*q11 + 2*x[0]*x[1]*q12 +  x[1]*x[1]*q22
return f

that the user calls with their parameters (q11, q12, q22) to get a function that can be used as a callback for the C optimization library.

(The example above is contrived and simplified, the whole point of doing this in Cython is that I want nearly-C-efficient callbacks to give to the library.)

There is no way to do this in C, as observed by people in my other question. But in Cython I can compile this:

cdef class Quadratic:
    cdef double q11
    cdef double q12
    cdef double q22
    def __cinit__(self, double a, double b, double c):
        self.q11 = a
        self.q12 = b
        self.q22 = c
    cdef f(self, double x[]):
        return self.q11*x[0]*x[0] + self.q12*x[0]*x[1] + self.q22*x[1]*x[1]

(I have not yet tried using a generated function like this as an input to the library).

My question - is there Python overhead in this evaluation? I would like to have the parameterized function be nearly as efficient as if it were written in C.

If that is possible, how does Cython achieve this?

(Unless this is a very new addition to Cython then) this is not valid Cython code and can't be made to work. It fails for me with the error

C function definition not allowed here

The reason for this is that - as you already know - it is a genuine limitation of C: there is no valid C code that Cython could generate to create a closure. The reason being is that C would need to be able to dynamically generate a function at runtime to access the closured variables, and that isn't possible in standard C.

There are (at least) ~~two~~ three ways round this. First (and probably most sensible) you change your interface to accept a data pointer:

double[] minimizer(ObjFun f, void* data)

with

typedef double (*ObjFun)(double x[], void* data)

(where minimizer passes data to f.) If you're using an external library then you can't make this change, but I'd be surprised if it didn't use something like this - it's the standard solution.

The second way is to use ctypes to convert a Python callable into a C function pointer. This gets round the problem by dynamically generating machine code at runtime to match the C calling convention (and so can be blocked by security features that prohibit making memory executable). I show a basic example of how to do this in the second half of this answer. (I'm sure I have another more thorough answer showing this but I can't find it right now).

Using ctypes involves a reasonable amount of Python overhead, which is unavoidable.

(Edited addition) The third option to emulate a closure (somewhat unsatisfactorily) is to store the data in global variables:

# at global scope
q11 = 1.2
q22 = 3.2
q33 = 4.0

cdef double f(double x[]):
    # as before

The big limitation is that you can only use one version of f at once, so you can't run things in parallel or anything like that, but given the constraints that might be acceptable.

With respect to your cdef class version: calling it is pretty efficient but it effectively has a signature of: