correct setup.py for mixing Python and C++

Question

I'm trying to mix both languages and I'm following the nice example provided by pybind here. I actually checked this post to improve on it so I can fall back to Python functions whenever a compiled function doesn't exist. The problem I have now is that my configure.py is not building the correct package. Let me develop: the structure of my code is something like this:

$ tree .
.
├── AUTHORS.md
├── CMakeLists.txt
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.md
├── conda.recipe
│   ├── bld.bat
│   └── ...
├── docs
│   ├── Makefile
│   └── ...
├── cmake_example
│   ├── __init__.py
│   ├── __main__.py
│   ├── geometry
│   │   ├── __init__.py
│   │   ├── triangle.py
│   │   └── ...
│   ├── quadrature
│   │   ├── __init__.py
│   │   ├── legendre
│   │   └── ...
│   └── utils
│       ├── __init__.py
│       ├── classes.py
│       └── ...
├── pybind11
│   ├── CMakeLists.txt
│   └── ...
├── setup.py
├── src
│   └── main.cpp
└── tests
    └── test.py

Where I put ellipsis to simplify the directory structure, but you can see there are a few modules. Now my setup.py file looks like this

import os
import re
import sys
import platform
import subprocess
import glob

from setuptools import setup, Extension, find_packages
from setuptools.command.build_ext import build_ext
from distutils.version import LooseVersion

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)    

class CMakeBuild(build_ext):
    def run(self):
        try:
            out = subprocess.check_output(['cmake', '--version'])
        except OSError:
            raise RuntimeError("CMake must be installed to build the following extensions: " +
                               ", ".join(e.name for e in self.extensions))

        if platform.system() == "Windows":
            cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
            if cmake_version < '3.1.0':
                raise RuntimeError("CMake >= 3.1.0 is required on Windows")

        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
        cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir,
                      '-DPYTHON_EXECUTABLE=' + sys.executable]

        cfg = 'Debug' if self.debug else 'Release'
        build_args = ['--config', cfg]

        if platform.system() == "Windows":
            cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(cfg.upper(), extdir)]
            if sys.maxsize > 2**32:
                cmake_args += ['-A', 'x64']
            build_args += ['--', '/m']
        else:
            cmake_args += ['-DCMAKE_BUILD_TYPE=' + cfg]
            build_args += ['--', '-j2']

        env = os.environ.copy()
        env['CXXFLAGS'] = '{} -DVERSION_INFO=\\"{}\\"'.format(env.get('CXXFLAGS', ''),
                                                              self.distribution.get_version())
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)
        subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)


kwargs = dict(
    name="cmake_example",
    ext_modules=[CMakeExtension('cmake_example._mymath')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
    packages='cmake_example',
)

# likely there are more exceptions
try:
    setup(**kwargs)
except subprocess.CalledProcessError:
    print("ERROR: Cannot compile C accelerator module, use pure python version")
    del kwargs['ext_modules']
    setup(**kwargs)

which I took from this post. When I try to build the wheel using python setup.py bdist_wheel, and then I install using pip install ., I can't use my code because it complains that packages are not found:

>>> import cmake_example
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/aaragon/Local/cmake_example/cmake_example/__init__.py", line 11, in <module>
    from .geometry import Triangle
ModuleNotFoundError: No module named 'cmake_example.geometry'

If I manually add in setup.py the list with packages=['cmake_example', cmake_example.geometry] then it works, but I don't think this is the right way to do it because it would be super hard to keep up with adding new modules. I saw somewhere I could replace that line and use setuptools's findpackages, but this function doesn't prepend the cmake_example to the module, so it still breaks. What's the correct way to do what I'm trying to do?

Answer 1

If I manually add in setup.py the list with packages=['cmake_example', cmake_example.geometry] then it works, but I don't think this is the right way to do it because it would be super hard to keep up with adding new modules.

Either you do it manually or when it's getting hard to keep up with adding new modules there's setuptools.find_packages. Use like:

from setuptools import setup, find_packages
setup(
    name="HelloWorld",
    version="0.1",
    packages=find_packages(),
)