Repeated single or multiple tests with Nose

问题:

Similar to this question, I'd like to have Nose run a test (or all tests) n times -- but not in parallel.

I have a few hundred tests in a project; some are some simple unit tests. Others are integration tests w/ some degree of concurrency. Frequently when debugging tests I want to "hit" a test harder; a bash loop works, but makes for a lot of cluttered output -- no more nice single "." for each passing test. Having the ability to beat on the selected tests for some number of trials seems like a natural thing to ask Nose to do, but I haven't found it anywhere in the docs.

What's the simplest way to get Nose to do this (other than a bash loop)?

回答1:

You can write a nose test as a generator, and nose will then run each function yielded:

def check_something(arg):
    # some test ...

def test_something():
    for arg in some_sequence:
        yield (check_something, arg)

Using nose-testconfig, you could make the number of test runs a command line argument:

from testconfig import config

# ...

def test_something():
    for n in range(int(config.get("runs", 1))):
        yield (check_something, arg)

Which you'd call from the command line with e.g.

$ nosetests --tc=runs:5

... for more than one run.

Alternatively (but also using nose-testconfig), you could write a decorator:

from functools import wraps
from testconfig import config

def multi(fn):
    @wraps(fn)
    def wrapper():
        for n in range(int(config.get("runs", 1))):
            fn()
    return wrapper

@multi
def test_something():
    # some test ...

And then, if you want to divide your tests into different groups, each with their own command line argument for the number of runs:

from functools import wraps
from testconfig import config

def multi(cmd_line_arg):
    def wrap(fn):
        @wraps(fn)
        def wrapper():
            for n in range(int(config.get(cmd_line_arg, 1))):
                fn()
        return wrapper
    return wrap

@multi("foo")
def test_something():
    # some test ...

@multi("bar")
def test_something_else():
    # some test ...

Which you can call like this:

$ nosetests --tc=foo:3 --tc=bar:7

回答2:

One way is in the test itself:

Change this:

class MyTest(unittest.TestCase):

  def test_once(self):
      ...

To this:

class MyTest(unittest.TestCase):

  def assert_once(self):
      ...

  def test_many(self):
      for _ in range(5):
          self.assert_once()

回答3:

You'll have to write a script to do this, but you can repeat the test names on the commandline X times.

nosetests testname testname testname testname testname testname testname

etc.

回答4:

Solution I ended up using is create sh script run_test.sh:

var=0
while $1; do
    ((var++))
    echo "*** RETRY $var"
done

Usage:

./run_test.sh "nosetests TestName"

It runs test infinitely but stops on first error.

回答5:

There should never be a reason to run a test more than once. It's important that your tests are deterministic (i.e. given the same state of the codebase, they always produce the same result.) If this isn't the case, then instead of running tests more than once, you should redesign the tests and/or code so that they are.

For example, one reason why tests fail intermittently is a race condition between the test and the code-under-test (CUT). In this circumstance, a naive response is to add a big 'voodoo sleep' to the test, to 'make sure' that the CUT is finished before the test starts asserting.

This is error-prone though, because if your CUT is slow for any reason (underpowered hardware, loaded box, busy database, etc) then it will fail sporadically. A better solution in this instance is to have your test wait for an event, rather than sleeping.

The event could be anything of your choosing. Sometimes, events you can use are already being generated (e.g. Javascript DOM events, the 'pageRendered' kind of events that Selenium tests can make use of.) Other times, it might be appropriate for you to add code to your CUT which raises an event when it's done (perhaps your architecture involves other components that are interested in events like this.)

Often though, you'll need to re-write the test such that it tries to detect whether your CUT is finished executing (e.g. does the output file exist yet?), and if not, sleeps for 50ms and then tries again. Eventually it will time out and fail, but only do this after a very long time (e.g. 100 times the expected execution time of your CUT)

Another approach is to design your CUT using 'onion/hexagonal/ports'n'adaptors' principles, which insists your business logic should be free of all external dependencies. This means that your business logic can be tested using plain old sub-millisecond unit tests, which never touch the network or filesystem. Once this is done, you need far fewer end-to-end system tests, because they are now serving just as integration tests, and don't need to try to manipulate every detail and edge-case of your business logic going through the UI. This approach will also yield big benefits in other areas, such as improved CUT design (reducing dependencies between components), tests are much easier to write, and the time taken to run the whole test suite is much reduced.

Using approaches like the above can entirely eliminate the problem of unreliable tests, and I'd recommend doing so, to improve not just your tests, but also your codebase, and your design abilities.