How to avoid redundancy between c++ and boost::pyt

Question

I'm adding a python module in a c++ code, using boost::python. The c++ project is documented with doxygen. I want to create a documentation for the python module but I don't know how not to be redundant like this :

#include <boost/python.hpp>
using namespace boost::python;

/** @brief Sum two integers
  * @param a an integer
  * @param b another integer
  * @return sum of integers
  */
int sum(int a, int b)
{
    return a+b;
}

BOOST_PYTHON_MODULE(pymodule)
{ 
    def("sum",&sum,args("a","b"),
        "Sum two integers.\n\n:param a: an integer\n:param b: another integer\n:returns: sum of integers");
};

Here I say the same thing in docstring and doxygen comments. Any ideas ?

Edit: The c++ doc isn't public and the python interface is a subset of c++.

Answer 1

The 5gon12eder's idea is to extract doxygen comments and substitute them into python docstrings. He proposed a solution with a python script.

Here's another one with a CMake script because I'm using it to build my project. I hope it can help people with the same problem :

set(FUNCTION "sum")
file(READ "pymodule.cpp.in" CONTENTS)

# To find the line with the flag
string(REGEX REPLACE "\n" ";" CONTENTS "${CONTENTS}")
list(FIND CONTENTS "// Docstring_${FUNCTION}" INDEX)

# To extract doxygen comments
math(EXPR INDEX "${INDEX}+1")
list(GET CONTENTS ${INDEX} LINE)
while(${LINE} MATCHES "@([a-z]+) (.*)")
  string(REGEX MATCH "@([a-z]+) (.*)" LINE "${LINE}")
  set(DOXY_COMMENTS ${DOXY_COMMENTS} ${LINE})
  math(EXPR INDEX "${INDEX}+1")
  list(GET CONTENTS ${INDEX} LINE)
endwhile()

# To convert doxygen comments into docstrings
foreach(LINE ${DOXY_COMMENTS})
  string(REGEX REPLACE "@brief " "" LINE "${LINE}")
  if("${LINE}" MATCHES "@param ([a-zA-Z0-9_]+) (.*)")
    set(LINE ":param ${CMAKE_MATCH_1}: ${CMAKE_MATCH_2}")
  endif()
  if ("${LINE}" MATCHES "@return (.+)")
    set(LINE ":returns: ${CMAKE_MATCH_1}")
  endif()
  set(DOCSTRING ${DOCSTRING} ${LINE})
endforeach()
string(REPLACE ";" "\\n" DOCSTRING "${DOCSTRING}")

# To insert docstrings in cpp file
set(Docstring_${FUNCTION} ${DOCSTRING})
configure_file("pymodule.cpp.in" "pymodule.cpp" @ONLY)

pymodule.cpp.in :

/**
 * @file pymodule.cpp
 */

#include<boost/python.hpp>
using namespace boost::python;

// Docstring_sum
/** @brief Sum two integers
  * @param a an integer
  * @param b another integer
  * @return sum of integers
  */
int sum(int a, int b) {
  return a+b;
}

BOOST_PYTHON_MODULE(pymodule){ 
  def("sum",&sum,args("a","b"),
      "@Docstring_sum@");
};

In this case, the script will generate pymodule.cpp with good docstring.

Answer 2

I'm a fan of code generation and I believe that this is a reasonable situation to deploy it.

If you're a bit disciplined in writing your Doxygen DocStrings and refrain from complicated markup in them, it is not that hard to write a small parser that extracts them and substitutes them back into the Python DocStrings.

Here is a small example. It won't be powerful enough to handle any realistic use-case but I believe that extending it would not be hard and worth the effort unless you only have a hand-full of functions to document.

Place a special comment before each Doxygen DocString that gives the following comment block a name. Here, I'm using the syntax

// DocString: sum
/**
 * @brief Sum two integers
 * @param a an integer
 * @param b another integer
 * @return sum of integers
 *
 */
int sum(int a, int b);

to associate the name sum with the following DocString.

Then, place another special string inside the Python bindings that references that name. I'm using the following syntax here.

BOOST_PYTHON_MODULE(pymodule)
{ 
  def("sum",&sum,args("a","b"), "@DocString(sum)");
};

Now we need a tool to extract the Doxygen DocStrings and substitutes them into the Python bindings.

As I've said, this example is contrived but it should show the idea and demonstrate that it is not too hard to do.

import re
import sys

def parse_doc_string(istr):
    pattern = re.compile(r'@(\w+)\s+(.*)')
    docstring = list()
    for line in map(lambda s : s.strip(), istr):
        if line == '/**':
            continue
        if line == '*/':
            return docstring
        line = line.lstrip('* ')
        match = pattern.match(line)
        if match:
            docstring.append((match.group(1), match.group(2)))

def extract(istr, docstrings):
    pattern = re.compile(r'^//\s*DocString:\s*(\w+)$')
    for line in map(lambda s : s.strip(), istr):
        match = pattern.match(line)
        if match:
            token = match.group(1)
            docstrings[token] = parse_doc_string(istr)

def format_doc_string(docstring):
    return '\n'.join('{}: {}'.format(k, v) for (k, v) in docstring)

def escape(string):
    return string.replace('\n', r'\n')

def substitute(istr, ostr, docstrings):
    pattern = re.compile(r'@DocString\((\w+)\)')
    for line in map(lambda s : s.rstrip(), istr):
        for match in pattern.finditer(line):
            token = match.group(1)
            docstring = format_doc_string(docstrings[token])
            line = line.replace(match.group(0), escape(docstring))
        print(line, file=ostr)

if __name__ == '__main__':
    sourcefile = sys.argv[1]
    docstrings = dict()
    with open(sourcefile) as istr:
        extract(istr, docstrings)
    with open(sourcefile) as istr:
        with sys.stdout as ostr:
            substitute(istr, ostr, docstrings)

Running this script over the source file produces the following output.

#include <boost/python.hpp>
using namespace boost::python;

// DocString: sum
/**
 * @brief Sum two integers
 * @param a an integer
 * @param b another integer
 * @return sum of integers
 *
 */
int sum(int a, int b)
{
  return a+b;
}

BOOST_PYTHON_MODULE(pymodule)
{
  def("sum",&sum,args("a","b"), "brief: Sum two integers\nparam: a an integer\nparam: b another integer\nreturn: sum of integers");
};

Add two hours of polishing to the script and you're good to go.

Since this is likely to be of interest to other people as well, I wouldn't be surprised if somebody had already written such a script. And if not, publishing yours as free software would certainly be welcomed by others.