Python CLI program unit testing

Question

I am working on a python Command-Line-Interface program, and I find it boring when doing testings, for example, here is the help information of the program:

usage: pyconv [-h] [-f ENCODING] [-t ENCODING] [-o file_path] file_path

Convert text file from one encoding to another.

positional arguments:
  file_path

optional arguments:
  -h, --help            show this help message and exit
  -f ENCODING, --from ENCODING
                        Encoding of source file
  -t ENCODING, --to ENCODING
                        Encoding you want
  -o file_path, --output file_path
                        Output file path

When I made changes on the program and want to test something, I must open a terminal, type the command(with options and arguments), type enter, and see if any error occurs while running. If error really occurs, I must go back to the editor and check the code from top to end, guessing where the bug positions, make small changes, write print lines, return to the terminal, run command again...

Recursively.

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?

Answer 1

This isn't for Python specifically, but what I do to test command-line scripts is to run them with various predetermined inputs and options and store the correct output in a file. Then, to test them when I make changes, I simply run the new script and pipe the output into diff correct_output -. If the files are the same, it outputs nothing. If they're different, it shows you where. This will only work if you are on Linux or OS X; on Windows, you will have to get MSYS.

Example:

python mycliprogram --someoption "some input" | diff correct_output -

To make it even easier, you can add all these test runs to your 'make test' Makefile target, which I assume you already have. ;)

If you are running many of these at once, you could make it a little more obvious where each one ends by adding a fail tag:

python mycliprogram --someoption "some input" | diff correct_output - || tput setaf 1 && echo "FAILED"

Answer 2

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?

The only difference is that when you run Python module as a script, its __name__ attribute is set to '__main__'. So generally, if you intend to run your script from command line it should have following form:

import sys

# function and class definitions, etc.
# ...
def foo(arg):
    pass

def main():
    """Entry point to the script"""

    # Do parsing of command line arguments and other stuff here. And then
    # make calls to whatever functions and classes that are defined in your
    # module. For example:
    foo(sys.argv[1])


if __name__ == '__main__':
    main()

Now there is no difference, how you would use it: as a script or as a module. So inside your unit-testing code you can just import foo function, call it and make any assertions you want.

Answer 3

The short answer is yes, you can use unit tests, and should. If your code is well structured, it should be quite easy to test each component separately, and if you need to to can always mock sys.argv to simulate running it with different arguments.

Answer 4

I would not test the program as a whole this is not a good test strategy and may not actually catch the actual spot of the error. The CLI interface is just front end to an API. You test the API via your unit tests and then when you make a change to a specific part you have a test case to exercise that change.

So, restructure your application so that you test the API and not the application it self. But, you can have a functional test that actually does run the full application and checks that the output is correct.

In short, yes testing the code is the same as testing any other code, but you must test the individual parts rather than their combination as a whole to ensure that your changes do not break everything.

Answer 5

I think it's perfectly fine to test functionally on a whole-program level. It's still possible to test one aspect/option per test. This way you can be sure that the program really works as a whole. Writing unit-tests usually means that you get to execute your tests quicker and that failures are usually easier to interpret/understand. But unit-tests are typically more tied to the program structure, requiring more refactoring effort when you internally change things.

Anyway, using py.test, here is a little example for testing a latin1 to utf8 conversion for pyconv::

# content of test_pyconv.py

import pytest

# we reuse a bit of pytest's own testing machinery, this should eventually come
# from a separatedly installable pytest-cli plugin. 
pytest_plugins = ["pytester"]

@pytest.fixture
def run(testdir):
    def do_run(*args):
        args = ["pyconv"] + list(args)
        return testdir._run(*args)
    return do_run

def test_pyconv_latin1_to_utf8(tmpdir, run):
    input = tmpdir.join("example.txt")
    content = unicode("\xc3\xa4\xc3\xb6", "latin1")
    with input.open("wb") as f:
        f.write(content.encode("latin1"))
    output = tmpdir.join("example.txt.utf8")
    result = run("-flatin1", "-tutf8", input, "-o", output)
    assert result.ret == 0
    with output.open("rb") as f:
        newcontent = f.read()
    assert content.encode("utf8") == newcontent

After installing pytest ("pip install pytest") you can run it like this::

$ py.test test_pyconv.py
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.4.5dev1
collected 1 items

test_pyconv.py .

========================= 1 passed in 0.40 seconds =========================

The example reuses some internal machinery of pytest's own testing by leveraging pytest's fixture mechanism, see http://pytest.org/latest/fixture.html. If you forget about the details for a moment, you can just work from the fact that "run" and "tmpdir" are provided for helping you to prepare and run tests. If you want to play, you can try to insert a failing assert-statement or simply "assert 0" and then look at the traceback or issue "py.test --pdb" to enter a python prompt.

Answer 6

You can use standard unittest module:

# python -m unittest <test module>

or use nose as a testing framework. Just write classic unittest files in separate directory and run:

# nosetests <test modules directory>

Writing unittests is easy. Just follow online manual for unittesting