How can I include special characters (tab, newline

Question

Given the following python script:

# dedupe.py
import re

def dedupe_whitespace(s,spacechars='\t '):
    """Merge repeated whitespace characters.
    Example:
    >>> dedupe_whitespace(r"Green\t\tGround")  # doctest: +REPORT_NDIFF
    'Green\tGround'
    """
    for w in spacechars:
        s = re.sub(r"("+w+"+)", w, s)
    return s

The function works as intended within the python interpreter:

$ python
>>> import dedupe
>>> dedupe.dedupe_whitespace('Purple\t\tHaze')
'Purple\tHaze'
>>> print dedupe.dedupe_whitespace('Blue\t\tSky')
Blue    Sky

However, the doctest example fails because tab characters are converted to spaces before comparison to the result string:

>>> import doctest, dedupe
>>> doctest.testmod(dedupe)

gives

Failed example:
    dedupe_whitespace(r"Green           Ground")  #doctest: +REPORT_NDIFF
Differences (ndiff with -expected +actual):
    - 'Green  Ground'
    ?       -
    + 'Green Ground'

How can I encode tab characters in a doctest heredoc string so that a test result comparison is performed appropriately?

Answer 1

You must set the NORMALIZE_WHITESPACE. Or, alternatively, capture the output and compare it to the expected value:

def dedupe_whitespace(s,spacechars='\t '):
    """Merge repeated whitespace characters.
    Example:
    >>> output = dedupe_whitespace(r"Black\t\tGround")  #doctest: +REPORT_NDIFF
    >>> output == 'Black\tGround'
    True
    """

From the doctest documentation section How are Docstring Examples Recognized?:

All hard tab characters are expanded to spaces, using 8-column tab stops. Tabs in output generated by the tested code are not modified. Because any hard tabs in the sample output are expanded, this means that if the code output includes hard tabs, the only way the doctest can pass is if the NORMALIZE_WHITESPACE option or directive is in effect. Alternatively, the test can be rewritten to capture the output and compare it to an expected value as part of the test. This handling of tabs in the source was arrived at through trial and error, and has proven to be the least error prone way of handling them. It is possible to use a different algorithm for handling tabs by writing a custom DocTestParser class.

Edit: My mistake, I understood the docs the other way around. Tabs are being expanded to 8 spaces at both the string argument passed to dedupe_whitespace and the string literal being compared on the next line, so output contains:

"Black Ground"

and is being compared to:

"Black        Ground"

I can't find a way to overcome this limitation without writing your own DocTestParser or testing for deduplicated spaces instead of tabs.

Answer 2

I've gotten this to work using literal string notation for the docstring:

def join_with_tab(iterable):
    r"""
    >>> join_with_tab(['1', '2'])
    '1\t2'
    """

    return '\t'.join(iterable)

if __name__ == "__main__":
    import doctest
    doctest.testmod()

Answer 3

I got it to work by escaping the tab character in the expected string:

>>> function_that_returns_tabbed_text()
'\\t\\t\\tsometext\\t\\t'

instead of

>>> function_that_returns_tabbed_text()
\t\t\tsometext\t\t

Answer 4

This is basically YatharhROCK's answer, but a bit more explicit. You can use raw strings or double escaping. But why?

You need the string literal to contain valid Python code that, when interpreted, is the code you want to run/test. These both work:

#!/usr/bin/env python

def split_raw(val, sep='\n'):
  r"""Split a string on newlines (by default).

  >>> split_raw('alpha\nbeta\ngamma')
  ['alpha', 'beta', 'gamma']
  """
  return val.split(sep)


def split_esc(val, sep='\n'):
  """Split a string on newlines (by default).

  >>> split_esc('alpha\\nbeta\\ngamma')
  ['alpha', 'beta', 'gamma']
  """
  return val.split(sep)

import doctest
doctest.testmod()

The effect of using raw strings and the effect of double-escaping (escape the slash) both leaves in the string two characters, the slash and the n. This code is passed to the Python interpreter, which takes "slash then n" to mean "newline character" inside a string literal.

Use whichever you prefer.

Answer 5

TL;DR: Escape the backslash, i.e., use \\n or \\t instead of \n or \t in your otherwise unmodified strings;

You probably don't want to make your docstrings raw as then you won't be able to use any Python string escapes including those you might want to.

For a method that supports using normal escapes, just escape the backslash in the backslash-character escape so after Python interprets it, it leaves a literal backslash followed by the character which doctest can parse.

Answer 6

It's the raw heredoc string notation (r""") that did the trick:

# filename: dedupe.py
import re,doctest
def dedupe_whitespace(s,spacechars='\t '):
    r"""Merge repeated whitespace characters.
    Example:
    >>> dedupe_whitespace('Black\t\tGround')  #doctest: +REPORT_NDIFF
    'Black\tGround'
    """
    for w in spacechars:
        s = re.sub(r"("+w+"+)", w, s)
    return s

if __name__ == "__main__":
    doctest.testmod()