How to organise python / Boost Python projects

Question

I have a python project, to which I would like to interface with some C++ libraries using Boost::Python. I would like to know how others go about organising their python/boost::python/C++ code within the same project.

By organisation I mean in terms of file/directory structure, build procedures etc.

Answer 1

In what follows, pif denotes Python InterFace. First I've got a generic header file, called conv_pif.hpp, which has Boost headers and C++ Std Library headers and such. Then for each boost python module, I have a file (here corresponding to the example module genocpp) of the form string_pif.cpp, where string corresponds roughly to the name of the module.

****************************************************************************************
geno_pif.cpp
****************************************************************************************
#include "conv_pif.hpp"
#include <boost/python.hpp>
#include "geno.hpp"

void export_cppvec_conv();

void export_geno()
{
  boost::python::def("write_geno_table_affy6_to_file", write_geno_table_affy6_to_file);
}

BOOST_PYTHON_MODULE(genocpp)
{
  export_geno();
  export_cppvec_conv();
}
*****************************************************************************************

The function export_cppvec_conv corresponds to a (templated) converter to/from C++ vectors to python lists. I have the actual converters in the file cppvec_conv_pif.cpp. In particular, this defines export_cppvec_conv, which uses template instantatiation, so I can get away without including it in geno_pif.cpp. For illustration, the contents of export_cppvec_conv are as follows, where cppvec_to_python_list and cppvec_from_python_list are defined in the body of cppvec_conv_pif.cpp.

******************************************
cppvec_conv_pif.cpp (extract)
******************************************
void export_cppvec_conv()
{
  boost::python::to_python_converter<vector<double>, cppvec_to_python_list<double> >();
  cppvec_from_python_list<double>();

  boost::python::to_python_converter<vector<int>, cppvec_to_python_list<int> >();
  cppvec_from_python_list<int>();

  boost::python::to_python_converter<vector<string>, cppvec_to_python_list<string> >();
  cppvec_from_python_list<string>();
}
******************************************

One can add as many converters as needed for the genocpp module. Then of course I've got the headers for the geno functions in geno.hpp. Finally, I have a Scons file which links everything together

******************************************
Sconstruct
******************************************
#!/usr/bin/python

import commands, glob, os

# Common file, for both executables and Python Interface
common_files = """geno print"""

def pyversion():
    pystr = commands.getoutput('python -V')
    version = pystr.split(' ')[1]
    major, minor = version.split('.')[:2]
    return major + '.' + minor

common_base = Split(common_files)
common = [f + ".cpp" for f in common_base]

# For Python interface only
pif_conv = Split("cppvec_conv cppmap_conv cppset_conv")
pif_conv_files = [t+"_pif.cpp" for t in pif_conv]

pif = Split("geno")
pif_files = [t+"_pif.cpp" for t in pif]

# Boost Python Environment
boost_python_env = Environment(
    CPPPATH=["/usr/include/python"+pyversion(), "."],
    CXXFLAGS='-ftemplate-depth-100 -fPIC -Wall -Werror -pedantic -pipe -O3 -ffast-math -march=opteron',
    #CXXFLAGS='-ftemplate-depth-100 -fPIC -Wall -pedantic -O0 -g',
    CPPDEFINES=['BOOST_PYTHON_DYNAMIC_LIB'],
    LIBPATH=["/usr/lib/python"+pyversion()+"/config"],
    LIBS=["python"+pyversion(), "m", "boost_python"],
    SHLIBPREFIX="", #gets rid of lib prefix
    SHOBJSUFFIX = ".bpo"
)

boost_python_env.SharedLibrary(target='genocpp', source = common + pif_conv_files + pif_files)

In this case, there is only one module, so pif_files just has geno_pif.cpp. Otherwise, I would select just those I want for the module. Hmm, maybe it would be easiest to just upload a working example somewhere. If anyone is interested in more detail, I guess I could edit this?

Regards, Faheem

Answer 2

I can't give you direct advice on this, but a package manager for Gentoo called paludis does this, and from what I know, its developers are very capable, so its sources might serve as a good example on how to do this.

I personally would however recommend against Boost Python. It is said to be very slow and memory-consuming compared to other binding tools such as cython, SWIG or SIP.