(Python-Sphinx) How to generate HTML Documentation

Question

I'm working on a Python project using PyCharm and now I need to generate the corresponding API documentation. I'm documenting the code methods and classes using docstrings. I read about Sphinx and Doxygen, with Sphinx being the most recommended right now. I tried to configure Sphinx whitin PyCharm but I had no luck in getting it working.

This is the project structure:

and this was the I/O interaction with the command Sphinx Quickstart

C:\Python\Python36\Scripts\sphinx-quickstart.exe
Welcome to the Sphinx 1.6.3 quickstart utility.

Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).

Enter the root path for documentation.
> Root path for the documentation [.]: 

You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]: 

Inside the root directory, two more directories will be created; "_templates"
for custom HTML templates and "_static" for custom stylesheets and other static
files. You can enter another prefix (such as ".") to replace the underscore.
> Name prefix for templates and static dir [_]: .

The project name will occur in several places in the built documentation.
> Project name: Attributed Graph Profiler
> Author name(s): M.C & D.A.T.

Sphinx has the notion of a "version" and a "release" for the
software. Each version can have multiple releases. For example, for
Python the version is something like 2.5 or 3.0, while the release is
something like 2.5.1 or 3.0a1.  If you don't need this dual structure,
just set both to the same value.
> Project version []: 0.0.1
> Project release [0.0.1]: 

If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.

For a list of supported codes, see
http://sphinx-doc.org/config.html#confval-language.
> Project language [en]: 

The file name suffix for source files. Commonly, this is either ".txt"
or ".rst".  Only files with this suffix are considered documents.
> Source file suffix [.rst]: 

One document is special in that it is considered the top node of the
"contents tree", that is, it is the root of the hierarchical structure
of the documents. Normally, this is "index", but if your "index"
document is a custom template, you can also set this to another filename.
> Name of your master document (without suffix) [index]: 

Sphinx can also add configuration for epub output:
> Do you want to use the epub builder (y/n) [n]: 

Please indicate if you want to use one of the following Sphinx extensions:
> autodoc: automatically insert docstrings from modules (y/n) [n]: y
> doctest: automatically test code snippets in doctest blocks (y/n) [n]: 
> intersphinx: link between Sphinx documentation of different projects (y/n) [n]: 
> todo: write "todo" entries that can be shown or hidden on build (y/n) [n]: 
> coverage: checks for documentation coverage (y/n) [n]: 
> imgmath: include math, rendered as PNG or SVG images (y/n) [n]: 
> mathjax: include math, rendered in the browser by MathJax (y/n) [n]: 
> ifconfig: conditional inclusion of content based on config values (y/n) [n]: 
> viewcode: include links to the source code of documented Python objects (y/n) [n]: y
> githubpages: create .nojekyll file to publish the document on GitHub pages (y/n) [n]: y

A Makefile and a Windows command file can be generated for you so that you
only have to run e.g. `make html' instead of invoking sphinx-build
directly.
> Create Makefile? (y/n) [y]: 
> Create Windows command file? (y/n) [y]: 

Creating file .\conf.py.
Creating file .\index.rst.
Creating file .\Makefile.
Creating file .\make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file .\index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.


Process finished with exit code 0

Then I moved to the /docs folder

, edited the conf.py file:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# "Query Rewriter" documentation build configuration file, created by
# sphinx-quickstart on Thu Sep 21 14:56:19 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.append(os.path.abspath("../../query_rewriter"))


# -- General configuration ------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc',
    'sphinx.ext.intersphinx',
    'sphinx.ext.ifconfig',
    'sphinx.ext.viewcode',
    'sphinx.ext.githubpages']

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'

# The master toctree document.
master_doc = 'index'

# General information about the project.
project = '"Query Rewriter"'
copyright = '2017, M.C & D.A.T'
author = 'M.C & D.A.T'

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '0.0.1'
# The full version, including alpha/beta/rc tags.
release = '0.0.1'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = "en"

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'

# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False


# -- Options for HTML output ----------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Theme options are theme-specific and customize the look and feel of a theme
# further.  For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# This is required for the alabaster theme
# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars
html_sidebars = {
    '**': [
        'about.html',
        'navigation.html',
        'relations.html',  # needs 'show_related': True theme option to display
        'searchbox.html',
        'donate.html',
    ]
}


# -- Options for HTMLHelp output ------------------------------------------

# Output file base name for HTML help builder.
htmlhelp_basename = 'QueryRewriterdoc'


# -- Options for LaTeX output ---------------------------------------------

latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    #
    # 'papersize': 'letterpaper',

    # The font size ('10pt', '11pt' or '12pt').
    #
    # 'pointsize': '10pt',

    # Additional stuff for the LaTeX preamble.
    #
    # 'preamble': '',

    # Latex figure (float) alignment
    #
    # 'figure_align': 'htbp',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
#  author, documentclass [howto, manual, or own class]).
latex_documents = [
    (master_doc, 'QueryRewriter.tex', '"Query Rewriter" Documentation',
     'M.C \\& D.A.T', 'manual'),
]


# -- Options for manual page output ---------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
    (master_doc, 'queryrewriter', '"Query Rewriter" Documentation',
     [author], 1)
]


# -- Options for Texinfo output -------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
#  dir menu entry, description, category)
texinfo_documents = [
    (master_doc, 'QueryRewriter', '"Query Rewriter" Documentation',
     author, 'QueryRewriter', 'One line description of project.',
     'Miscellaneous'),
]




# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}

and ran the following command:

B:\_Python_Workspace\AttributedGraphProfiler\docs>make html
Running Sphinx v1.6.3
making output directory...
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in .build\html.

B:\_Python_Workspace\AttributedGraphProfiler\docs>

I thought I was done, but this is the poor result I got, without any documentation for classes and modules.

index.html

genindex.html

Am I doing something wrong? Thanks in advance for your time.

Answer 1

Just solved excatly the same problem Juan. Sphinx unfortunately is not a fully automated doc generator from code comments like doxygen, jautodoc etc. As in the link mentioned in mzjn's comment some steps are necessary for a proper work.

As I see you are working on Pycharm, so I will touch on Pycharm-Sphinx integration. I hope you will not have to change anything manually like conf.py.

1. In PyCharm "File/setting/tools/python integrated tools" define the sphinx-working-directory to codebase/Docs. (Just for clearness, choose where ever you want)

   So your sphinx scripts will run at this path.

---

2. Run "Tools/Sphinx Quickstart" in PyCharm

   Like you wrote above select the proper options. But "autodoc" is a must (y) and "Separate source and build directories" is recommended (y) to understand what is going on. This script will generate the skeleton of the sphinx project.

---

3. Create a python task in Run/Edit-Configurations... in PyCharm like below.

   Be careful with the python interpreter and your script (If you use python environment like me). This script will generate the rst files for your modules.

   source/ shows Docs/Source directory created by 1.step. It has the .rst files for our modules.

   ../ shows our modules' py files.

UPDATE 1:

A-) Run this task to generate the rst files.

B-) Add "modules" term to index.rst file, like;

bla bla
.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules
bla bla

There is no need to run and add "modules" term again in every doc creation. Step A is necessary only when new modules are introduced in the project.

4. Create a Python Docs task in Run/Edit-Configurations... in PyCharm like below.

   command shows the documentation type.

   Input shows our rst files path.

   output shows our document output directory. (FINALLY :))

5. Run and enjoy...

UPDATE 2:

If you integrate with the Readthedocs: (After sign up and github authorization)

1. You have to add modules path to sys.path. Readthedocs calls sphinx-build so it should know the path of the py files. Basically uncomment the line in conf.py

   sys.path.insert(0, os.path.abspath('..//..//')) (if working directory is pyfiles/Docs/source)

2. Always check the build results on readthedocs, siply click on lines like these: You will see if there is an error or a warning.
3. If you didn't do 1 & 2 probably you will not have a working "Indices and tables" section. genindex.html and py-modindex.html will be respectively an empty template and a non existing page.

Answer 2

Prior to make html you should do the reading the apidoc comments from your code and creating .rst files. You should run something like this from your project root folder:

sphinx-apidoc . -o ./docs -f tests

This will rewrite .rst files (hence -f) in docs folder and ignore to do the apidoc reading in tests package.

More on sphinx-apidoc you can find here.

After this there goes make html command.