Build Boost with multiple Python versions

Question

I use several versions of Python on my computer : 2.6.6, 2.7.6 and 2.7.9. When I compile Boost with boost-python, I have to give the Python to use in argument. If I want compatibility, have I to compile Boost for each Python version ? (it's quite huge !) Or is there a way to use only one build of Boost with several python versions ?

Answer 1

The official Python development cycle does not describe the stability of the application binary interface (ABI) between releases. For releases before Python 3.2, there is no guarantee for ABI compatibility. For 3.2 and beyond, PEP 384 defines the stable ABI where a subset of the Python/C API is guaranteed to maintain ABI compatibility. To use this subset, Py_LIMITED_API needs to be defined when building Boost.Python and extension modules.

While it primarily depends on the Python/C API types and functionality being used directly in user code or through Boost.Python, in general:

• for major-level releases, such as Python 2 and Python 3, Boost.Python and user code will need to be recompiled
• for minor-level releases, such as Python 2.6 and Python 2.7, Boost.Python and user code may need to be recompiled
• for micro-level releases, Boost.Python and user code rarely needs recompiled

In all cases where Boost.Python needs to be recompiled, no other Boost library should need to be recompiled. When building multiple versions of Boost.Python, verify a clean build occurs. Without a clean build, Boost.Python may build, but fail to properly link. For example, the PyClass_Type symbol should not be referenced in Python 3 Boost.Python builds, but without a clean build, previous build artifacts may populate the library:

$ ./bootstrap.sh --with-python=/usr/bin/python2
...
Detecting Python version... 2.7
$ ./b2 --with-python --buildid=2 # produces libboost_python-2.so
$ ./bootstrap.sh --with-python=/usr/bin/python3 --with-python-root=/usr
...
Detecting Python version... 3.3
$ ./b2 --with-python --buildid=3noclean # produces libboost_python-3noclean.so
$ ./b2 --with-python --clean
$ ./b2 --with-python --buildid=3 # produces libboost_python-3.so

$ nm -D stage/lib/libboost_python-2.so | grep PyClass_Type
                 U PyClass_Type
$ nm -D stage/lib/libboost_python-3noclean.so | grep PyClass_Type
                 U PyClass_Type
$ nm -D stage/lib/libboost_python-3.so | grep PyClass_Type

Note that even though the 3noclean build was built against Python 3, the previous build that was using Python 2 had artifacts that polluted the 3noclean library. Also, be aware that Boost.Python and user code may need recompiled to match the Python's CPU architecture and UCS-2 or UCS-4 unicode configurations.

Answer 2

As of Boost 1.67, you can build boost so it generates a separate version of libboost_python for each version of Python that you specify without having to clean or rebuild as part of the process.

For instance, I edit tools/build/src/user-config.jam to contain the versions of Python on the system:

using python : 2.7 : /opt/python/cp27-cp27mu/bin/python : /opt/python/cp27-cp27mu/include/python2.7 : /opt/python/cp27-cp27mu/lib ;
using python : 3.5 : /opt/python/cp35-cp35m/bin/python : /opt/python/cp35-cp35m/include/python3.5m : /opt/python/cp35-cp35m/lib ;
using python : 3.6 : /opt/python/cp36-cp36m/bin/python : /opt/python/cp36-cp36m/include/python3.6m : /opt/python/cp36-cp36m/lib ;
using python : 3.7 : /opt/python/cp37-cp37m/bin/python : /opt/python/cp37-cp37m/include/python3.7m : /opt/python/cp37-cp37m/lib ;

Then, I run ./b2 python=2.7,3.5,3.6,3.7 for the build step.

Some downstream packages expect the boost python libraries to be called libboost_python.so for Python 2.x and libboost_python3.so for Python 3.x, and you might need to explicitly specify -lboost_python27, -lboost_python35, etc. to get those programs to link properly.