How do I use Sphinx with Cython?

Question

I have recently Cythonized a project of mine by renaming all of the modules (except for the top-level __init__.py) to *.pyx, and by putting ext_modules = [Extension('foo', ['foo.pyx'])] in setup.py. Building and installing works fine. However, when I do cd doc; make html, Sphinx fails because it cannot import any of the modules which are now *.pyx.

If I edit doc/conf.py and change sys.path.insert(0, os.path.abspath('..')) to sys.path.insert(0, os.path.abspath('../build/temp.linux-x86_64-2.7')), then Sphinx can find all of the modules and can generate documentation, but in that case I get errors like error while formatting arguments for foo.bar: <built-in function bar> is not a Python function. Presumably this is because now Sphinx only has access to the *.so files, not the source files. That same sys.path modification also allows running the doctests via Sphinx (make doctest).

The other solution I tried was using the extension *.py instead of *.pyx (and using ext_modules = [Extension('foo', ['foo.py'])] in setup.py). In this case, the documentation builds correctly, but I think the doctests now bypass Cython.

I have not been able to find any information online regarding using Sphinx and Cython together. I have looked at source code for some projects which use both, but they don't seem to make use of docstrings in the *.pyx files. I know that Sage does, but that project is too complicated for me to pick apart.

Does Sphinx support docstrings in Cython files? If so, how do I make this work?

Answer 1

Feel free to leave a better answer, but here is a fix that I have found.

The dipy project manually imports their own module from doc/conf.py. This requires that the module first be installed, but it fixes the import errors (and doctests will run on the Cythonized files).

However, the error while formatting arguments problem is still there. First you need to instruct Cython to embed the method/function signatures into the *.so files. Do this by setting the embedsignature Cython directive. The dipy project sets this in each *.pyx file, but it is also possible to set it in setup.py (see Cython documentation for how to do that). This still doesn't put the method signatures into the Sphinx documentation though! There is a bug report and patch for the method signatures problem here. It is still not included in the latest Sphinx release as of now (1.1.3) but if you install Sphinx from the development repo it will work.

Answer 2

As Golgauth explains, Sphinx's autodoc module takes the docstrings from the .so, not the .pyx. The simplest way of generating your documentation without having to make any changes to your Sphinx configuration when cythonizing a Python module is to simple build the extension modules in place before you generate the docs:

python setup.py build_ext --inplace

That way autodoc will find the extension modules alongside the regular Python modules and will be able to generate the documentation as you'd expect.

To not risk forgetting this step you can edit the Makefile generated by sphinx-quickstart to build the extension modules prior to running sphinx-build:

html:
  @cd /path/to/setup.py; python setup.py build_ext --inplace
  ...

Answer 3

You look a litte bit confused here. Sphinx is not really a syntactic analyzer. Your Python code has to be runnable to make Sphinx able to catch the docstrings. That is why renaming the extensions files to ".py" doesn't help.

Well, I've been working with Sphinx and Cython recently, and would like to share my experience... Here's the full detailed procedure to get the automated generation of an html documentation for a given compiled Cython extension from docstrings :

[Note : I used Sphinx 1.1.3 and Cython 0.17.4]

First of all, use the Python's "docstrings" (with all the limitations it can have - by example, you can't describe a constructor. See docstrings specifications) in your Cython code :

cdef class PyLabNode:
    """
    This is a LabNode !!!
    """
    cdef LabNode* thisptr
    cdef PyLabNetwork network

    def __cinit__(self):
       self.thisptr = new LabNode()

    def __dealloc__(self):
       if self.thisptr:
           del self.thisptr

    def SetNetwork(self, PyLabNetwork net):
        """
        Set the network !!!
        """
        self.network = net

And recompile "yourextension.so".

Then run "sphinx-quickstart" and answer the questions. Do not forget to say yes when asked for "autodoc". This will generate the "Makefile", the "index.rst" file and the "conf.py" files.

This last "conf.py" has to be edited to tell Sphinx were to find your module :

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('../../parent/dir/of/yourextension/'))

The "index.rst" file has to be edited as well to tell which module might be analyzed :

Contents:

.. toctree::
   :maxdepth: 2


.. automodule:: yourextension
   :members:
   :undoc-members:
   :show-inheritance:

Finally build the documentation by doing :

$ make html

That was enough for me (I got the resulting set of html files in a ".../_build/html/" directory). May be Sphinx and Cython have evolved since the previous question was asked, but I had no "signature" issues to deal with. No particular Cython directive to use, nor any fix to apply to Sphinx...

Hope this helps.

EDIT : Well, I would like to take my words back. I encountered the issue "Dan" was mentioning concerning the "embedsignature" matter while using Epydoc (so I suppose this is an issue with Sphinx as well). Activating this compiler directive doesn't send python compliant signatures anyway :

PyLabNode.SetNetwork(self, PyLabNetwork net)

This has 2 drawback : The dotted notation for the class prefix and the typed parameter.

At the end, the only way I could figure out to send correct ones was to write a compliant signature at the very first line of the doc strings like so :

def SetNetwork(self, PyLabNetwork net):
    """
    SetNetwork(self, net)
    Set the net !!!
    @param self: Handler to this.
    @type self: L{PyLabNode}
    @param net: The network this node belongs to.
    @type net: L{PyLabNetwork}
    """
    self.network = net

Hope this can help both Sphinx and Epydoc users...

---

EDIT : Concerning the __cinit__, I was able to generate the doc successfully with Epidoc (didn't try with Sphinx) by doubling the description, like this:

# For Epydoc only (only used for docstring)
def __init__(self, sim):
    """
    __init__(self, sim)
    Constructor.
    @param sim: The simulator this binding is attached to.
    @type sim: L{PyLabSimulatorBase} 
    """ 

# Real Cython init
def __cinit__(self, PyLabSimulatorBase sim):
   self.thisptr = new LabNetBinding()
   self.sites = []
   simulator = sim