distribution : how to build a compiled module for

Question

I have build a python 3 module for my own process.

I use cython to compile and wrap C++ sources.

I have a linux (Debian Jessie) machine with python 3.4 and so cythonize make me a Processing.cpython-34m.so and copy it to /usr/local/lib/python3.4/dist-packages.

But when I use it on another machine which has python3.5, I have to recompile everything.

How can I build a linux or pip package from my machine for all python 3 version and multiple platforms (here, just Jessie and Stretch, which might be very closed indeed equal) ? Preferably without having to install all version of python 3 on my machine.

here is my setup.py file for cythonization :

from distutils.core import setup, Extension
from Cython.Build import cythonize

setup(ext_modules = cythonize(Extension(
            "MyProcessing",
            sources=["MyProcessing.pyx", "myprocess.cpp", "mythirdp.cpp"],
            language="c++", 
        )))

Thanks.

Answer 1

Python wheels are tied to the python ABI. The ABI is stable within a minor release, which is why your wheels have 34 tag - the typical solution is to create a wheel for each minor version of python you want to support, so you'd need to install all those versions. For more see PEP 425

For python 3.2+ there is a definition of a 'stable ABI' that works across all versions. I'm not sure exactly what the limitations are and to my knowledge cython doesn't work with it, but I believe for a hand written extension you could create a wheel that works across all versions of python 3.

Away from cython, if you're primarily wrapping c-code another potential solution is CFFI. It does not link against libpython, so I think you can generate a wheel that is version independent. It's not something I've used but Armin Ronacher has a package that seems to directly address this case https://github.com/getsentry/milksnake

edit: make sure to read @danny's answer about manylinux, that's an important point I forgot

Answer 2

Make manylinux1 binary wheels that work across distributions - see PEP-513

It involves running a docker build on the official PyPa manylinux1 docker images that builds binary wheels for all python versions.

These wheels can be distributed on PyPi and are usable across distributions.

The constraint is that the build needs to be done in a Centos5 distribution which the manylinux1 image is based on for compatibility.

See PyPa's manylinux demo repository for an example.