Python unittest: how to run only part of a test fi

Question

I have a test file that contains tests taking quite a lot of time (they send calculations to a cluster and wait for the result). All of these are in specific TestCase class.

Since they take time and furthermore are not likely to break, I'd want to be able to choose whether this subset of tests does or doesn't run (the best way would be with a command-line argument, ie "./tests.py --offline" or something like that), so I could run most of the tests often and quickly and the whole set once in a while, when I have time.

For now, I just use unittest.main() to start the tests.

Thanks.

Answer 1

Since you use unittest.main() you can just run python tests.py --help to get the documentation:

Usage: tests.py [options] [test] [...]

Options:
  -h, --help       Show this message
  -v, --verbose    Verbose output
  -q, --quiet      Minimal output
  -f, --failfast   Stop on first failure
  -c, --catch      Catch control-C and display results
  -b, --buffer     Buffer stdout and stderr during test runs

Examples:
  tests.py                               - run default set of tests
  tests.py MyTestSuite                   - run suite 'MyTestSuite'
  tests.py MyTestCase.testSomething      - run MyTestCase.testSomething
  tests.py MyTestCase                    - run all 'test*' test methods
                                               in MyTestCase

That is, you can simply do

python tests.py TestClass.test_method

Answer 2

The default unittest.main() uses the default test loader to make a TestSuite out of the module in which main is running.

You don't have to use this default behavior.

You can, for example, make three unittest.TestSuite instances.

1. The "fast" subset.

   fast = TestSuite()
   fast.addTests( TestFastThis )
   fast.addTests( TestFastThat )
   

2. The "slow" subset.

   slow = TestSuite()
   slow.addTests( TestSlowAnother )
   slow.addTests( TestSlowSomeMore )
   

3. The "whole" set.

   alltests = unittest.TestSuite([fast, slow])
   

Note that I've adjusted the TestCase names to indicate Fast vs. Slow. You can subclass unittest.TestLoader to parse the names of classes and create multiple loaders.

Then your main program can parse command-line arguments with optparse or argparse (available since 2.7 or 3.2) to pick which suite you want to run, fast, slow or all.

Or, you can trust that sys.argv[1] is one of three values and use something as simple as this

if __name__ == "__main__":
    suite = eval(sys.argv[1])  # Be careful with this line!
    unittest.TextTestRunner().run(suite)

Answer 3

Look into using a dedicated testrunner, like py.test, nose or possibly even zope.testing. They all have command line options for selecting tests.

Look for example as Nose: https://pypi.python.org/pypi/nose/1.3.0

Answer 4

This is the only thing that worked for me.

if __name__ == '__main__':
unittest.main( argv=sys.argv, testRunner = unittest.TextTestRunner(verbosity=2))

When I called it though I had to pass in the name of the class and test name. A little inconvenient since I don't have class and test name combination memorized.

python ./tests.py class_Name.test_30311

Removing the Class name and test name runs all the tests in your file. I find this MUCH more easier to deal with then the built in method since I don't really change my command on the CLI. Just add the parameter.

Enjoy, Keith

Answer 5

I have found another way to select the test_* methods that I only want to run by adding an attribute to them. You basically use a metaclass to decorate the callables inside the TestCase class that have the StepDebug attribute with a unittest.skip decorator. More info on

Skipping all unit tests but one in Python by using decorators and metaclasses

I don't know if it is a better solution than those above I am just providing it as an option.

Answer 6

Haven't found a nice way to do this before, so sharing here.

Goal: Get a set of test files together so they can be run as a unit, but we can still select any one of them to run by itself.

Problem: the discover method does not allow easy selection of a single test case to run.

Design: see below. This flattens the namespace so can select by TestCase class name, and leave off the the "tests1.test_core" prefix:

./run-tests TestCore.test_fmap

Code

  test_module_names = [
    'tests1.test_core',
    'tests2.test_other',
    'tests3.test_foo',
    ]

  loader = unittest.defaultTestLoader
  if args:
    alltests = unittest.TestSuite()
    for a in args:
      for m in test_module_names:
        try:
          alltests.addTest( loader.loadTestsFromName( m+'.'+a ) )
        except AttributeError as e:
          continue
  else:
    alltests = loader.loadTestsFromNames( test_module_names )

  runner = unittest.TextTestRunner( verbosity = opt.verbose )
  runner.run( alltests )