Getting Python's unittest results in a tearDow

Question

Is it possible to get the results of a test (i.e. whether all assertions have passed) in a tearDown() method? I'm running Selenium scripts, and I'd like to do some reporting from inside tearDown(), however I don't know if this is possible.

Answer 1

Inspired by scoffey’s answer, I decided to take mercilessnes to the next level, and have come up with the following.

It works in both vanilla unittest, and also when run via nosetests, and also works in Python versions 2.7, 3.2, 3.3, and 3.4 (I did not specifically test 3.0, 3.1, or 3.5, as I don’t have these installed at the moment, but if I read the source code correctly, it should work in 3.5 as well):

#! /usr/bin/env python

from __future__ import unicode_literals
import logging
import os
import sys
import unittest


# Log file to see squawks during testing
formatter = logging.Formatter(fmt='%(levelname)-8s %(name)s: %(message)s')
log_file = os.path.splitext(os.path.abspath(__file__))[0] + '.log'
handler = logging.FileHandler(log_file)
handler.setFormatter(formatter)
logging.root.addHandler(handler)
logging.root.setLevel(logging.DEBUG)
log = logging.getLogger(__name__)


PY = tuple(sys.version_info)[:3]


class SmartTestCase(unittest.TestCase):

    """Knows its state (pass/fail/error) by the time its tearDown is called."""

    def run(self, result):
        # Store the result on the class so tearDown can behave appropriately
        self.result = result.result if hasattr(result, 'result') else result
        if PY >= (3, 4, 0):
            self._feedErrorsToResultEarly = self._feedErrorsToResult
            self._feedErrorsToResult = lambda *args, **kwargs: None  # no-op
        super(SmartTestCase, self).run(result)

    @property
    def errored(self):
        if (3, 0, 0) <= PY < (3, 4, 0):
            return bool(self._outcomeForDoCleanups.errors)
        return self.id() in [case.id() for case, _ in self.result.errors]

    @property
    def failed(self):
        if (3, 0, 0) <= PY < (3, 4, 0):
            return bool(self._outcomeForDoCleanups.failures)
        return self.id() in [case.id() for case, _ in self.result.failures]

    @property
    def passed(self):
        return not (self.errored or self.failed)

    def tearDown(self):
        if PY >= (3, 4, 0):
            self._feedErrorsToResultEarly(self.result, self._outcome.errors)


class TestClass(SmartTestCase):

    def test_1(self):
        self.assertTrue(True)

    def test_2(self):
        self.assertFalse(True)

    def test_3(self):
        self.assertFalse(False)

    def test_4(self):
        self.assertTrue(False)

    def test_5(self):
        self.assertHerp('Derp')

    def tearDown(self):
        super(TestClass, self).tearDown()
        log.critical('---- RUNNING {} ... -----'.format(self.id()))
        if self.errored:
            log.critical('----- ERRORED -----')
        elif self.failed:
            log.critical('----- FAILED -----')
        else:
            log.critical('----- PASSED -----')


if __name__ == '__main__':
    unittest.main()

When run with unittest:

$ ./test.py -v
test_1 (__main__.TestClass) ... ok
test_2 (__main__.TestClass) ... FAIL
test_3 (__main__.TestClass) ... ok
test_4 (__main__.TestClass) ... FAIL
test_5 (__main__.TestClass) ... ERROR
[…]

$ cat ./test.log
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_1 ... -----
CRITICAL __main__: ----- PASSED -----
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_2 ... -----
CRITICAL __main__: ----- FAILED -----
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_3 ... -----
CRITICAL __main__: ----- PASSED -----
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_4 ... -----
CRITICAL __main__: ----- FAILED -----
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_5 ... -----
CRITICAL __main__: ----- ERRORED -----

When run with nosetests:

$ nosetests ./test.py -v
test_1 (test.TestClass) ... ok
test_2 (test.TestClass) ... FAIL
test_3 (test.TestClass) ... ok
test_4 (test.TestClass) ... FAIL
test_5 (test.TestClass) ... ERROR

$ cat ./test.log
CRITICAL test: ---- RUNNING test.TestClass.test_1 ... -----
CRITICAL test: ----- PASSED -----
CRITICAL test: ---- RUNNING test.TestClass.test_2 ... -----
CRITICAL test: ----- FAILED -----
CRITICAL test: ---- RUNNING test.TestClass.test_3 ... -----
CRITICAL test: ----- PASSED -----
CRITICAL test: ---- RUNNING test.TestClass.test_4 ... -----
CRITICAL test: ----- FAILED -----
CRITICAL test: ---- RUNNING test.TestClass.test_5 ... -----
CRITICAL test: ----- ERRORED -----

Background

I started with this:

class SmartTestCase(unittest.TestCase):

    """Knows its state (pass/fail/error) by the time its tearDown is called."""

    def run(self, result):
        # Store the result on the class so tearDown can behave appropriately
        self.result = result.result if hasattr(result, 'result') else result
        super(SmartTestCase, self).run(result)

    @property
    def errored(self):
        return self.id() in [case.id() for case, _ in self.result.errors]

    @property
    def failed(self):
        return self.id() in [case.id() for case, _ in self.result.failures]

    @property
    def passed(self):
        return not (self.errored or self.failed)

However, this only works in Python 2. In Python 3, up to and including 3.3, the control flow appears to have changed a bit: Python 3’s unittest package processes results after calling each test’s tearDown() method… this behavior can be confirmed if we simply add an extra line (or six) to our test class:

@@ -63,6 +63,12 @@
             log.critical('----- FAILED -----')
         else:
             log.critical('----- PASSED -----')
+        log.warning(
+            'ERRORS THUS FAR:\n'
+            + '\n'.join(tc.id() for tc, _ in self.result.errors))
+        log.warning(
+            'FAILURES THUS FAR:\n'
+            + '\n'.join(tc.id() for tc, _ in self.result.failures))


 if __name__ == '__main__':

Then just re-run the tests:

$ python3.3 ./test.py -v
test_1 (__main__.TestClass) ... ok
test_2 (__main__.TestClass) ... FAIL
test_3 (__main__.TestClass) ... ok
test_4 (__main__.TestClass) ... FAIL
test_5 (__main__.TestClass) ... ERROR
[…]

…and you will see that you get this as a result:

CRITICAL __main__: ---- RUNNING __main__.TestClass.test_1 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:

CRITICAL __main__: ---- RUNNING __main__.TestClass.test_2 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:

CRITICAL __main__: ---- RUNNING __main__.TestClass.test_3 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_4 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_5 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
__main__.TestClass.test_4

Now, compare the above to Python 2’s output:

CRITICAL __main__: ---- RUNNING __main__.TestClass.test_1 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:

CRITICAL __main__: ---- RUNNING __main__.TestClass.test_2 ... -----
CRITICAL __main__: ----- FAILED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_3 ... -----
CRITICAL __main__: ----- PASSED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_4 ... -----
CRITICAL __main__: ----- FAILED -----
WARNING  __main__: ERRORS THUS FAR:

WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
__main__.TestClass.test_4
CRITICAL __main__: ---- RUNNING __main__.TestClass.test_5 ... -----
CRITICAL __main__: ----- ERRORED -----
WARNING  __main__: ERRORS THUS FAR:
__main__.TestClass.test_5
WARNING  __main__: FAILURES THUS FAR:
__main__.TestClass.test_2
__main__.TestClass.test_4

Since Python 3 processes errors/failures after the test is torn down, we can’t readily infer the result of a test using result.errors or result.failures in every case. (I think it probably makes more sense architecturally to process a test’s results after tearing it down, however, it does make the perfectly valid use-case of following a different end-of-test procedure depending on a test’s pass/fail status a bit harder to meet…)

Therefore, instead of relying on the overall result object, instead we can reference _outcomeForDoCleanups as others have already mentioned, which contains the result object for the currently running test, and has the necessary errors and failrues attributes, which we can use to infer a test’s status by the time tearDown() has been called:

@@ -3,6 +3,7 @@
 from __future__ import unicode_literals
 import logging
 import os
+import sys
 import unittest


@@ -16,6 +17,9 @@
 log = logging.getLogger(__name__)


+PY = tuple(sys.version_info)[:3]
+
+
 class SmartTestCase(unittest.TestCase):

     """Knows its state (pass/fail/error) by the time its tearDown is called."""
@@ -27,10 +31,14 @@

     @property
     def errored(self):
+        if PY >= (3, 0, 0):
+            return bool(self._outcomeForDoCleanups.errors)
         return self.id() in [case.id() for case, _ in self.result.errors]

     @property
     def failed(self):
+        if PY >= (3, 0, 0):
+            return bool(self._outcomeForDoCleanups.failures)
         return self.id() in [case.id() for case, _ in self.result.failures]

     @property

This adds support for the early versions of Python 3.

As of Python 3.4, however, this private member variable no longer exists, and instead, a new (albeit also private) method was added: _feedErrorsToResult.

This means that for versions 3.4 (and later), if the need is great enough, one can — very hackishly — force one’s way in to make it all work again like it did in version 2…

@@ -27,17 +27,20 @@
     def run(self, result):
         # Store the result on the class so tearDown can behave appropriately
         self.result = result.result if hasattr(result, 'result') else result
+        if PY >= (3, 4, 0):
+            self._feedErrorsToResultEarly = self._feedErrorsToResult
+            self._feedErrorsToResult = lambda *args, **kwargs: None  # no-op
         super(SmartTestCase, self).run(result)

     @property
     def errored(self):
-        if PY >= (3, 0, 0):
+        if (3, 0, 0) <= PY < (3, 4, 0):
             return bool(self._outcomeForDoCleanups.errors)
         return self.id() in [case.id() for case, _ in self.result.errors]

     @property
     def failed(self):
-        if PY >= (3, 0, 0):
+        if (3, 0, 0) <= PY < (3, 4, 0):
             return bool(self._outcomeForDoCleanups.failures)
         return self.id() in [case.id() for case, _ in self.result.failures]

@@ -45,6 +48,10 @@
     def passed(self):
         return not (self.errored or self.failed)

+    def tearDown(self):
+        if PY >= (3, 4, 0):
+            self._feedErrorsToResultEarly(self.result, self._outcome.errors)
+

 class TestClass(SmartTestCase):

@@ -64,6 +71,7 @@
         self.assertHerp('Derp')

     def tearDown(self):
+        super(TestClass, self).tearDown()
         log.critical('---- RUNNING {} ... -----'.format(self.id()))
         if self.errored:
             log.critical('----- ERRORED -----')

…provided, of course, all consumers of this class remember to super(…, self).tearDown() in their respective tearDown methods…

Disclaimer: Purely educational, don’t try this at home, etc. etc. etc. I’m not particularly proud of this solution, but it seems to work well enough for the time being, and is the best I could hack up after fiddling for an hour or two on a Saturday afternoon…

Answer 2

CAVEAT: I have no way of double checking the following theory at the moment, being away from a dev box. So this may be a shot in the dark.

Perhaps you could check the return value of sys.exc_info() inside your tearDown() method, if it returns (None, None, None), you know the test case succeeded. Otherwise, you could use returned tuple to interrogate the exception object.

See sys.exc_info documentation.

Another more explicit approach is to write a method decorator that you could slap onto all your test case methods that require this special handling. This decorator can intercept assertion exceptions and based on that modify some state in self allowing your tearDown method to learn what's up.

@assertion_tracker
def test_foo(self):
    # some test logic

Answer 3

Python 2.7.

You can also get result after unittest.main():

t = unittest.main(exit=False)
print t.result

or use suite:

suite.addTests(tests)
result = unittest.result.TestResult()
suite.run(result)
print result

Answer 4

If you take a look at the implementation of unittest.TestCase.run, you can see that all test results are collected in the result object (typically a unittest.TestResult instance) passed as argument. No result status is left in the unittest.TestCase object.

So there isn't much you can do in the unittest.TestCase.tearDown method unless you mercilessly break the elegant decoupling of test cases and test results with something like this:

import unittest

class MyTest(unittest.TestCase):

    currentResult = None # holds last result object passed to run method

    def setUp(self):
        pass

    def tearDown(self):
        ok = self.currentResult.wasSuccessful()
        errors = self.currentResult.errors
        failures = self.currentResult.failures
        print ' All tests passed so far!' if ok else \
                ' %d errors and %d failures so far' % \
                (len(errors), len(failures))

    def run(self, result=None):
        self.currentResult = result # remember result for use in tearDown
        unittest.TestCase.run(self, result) # call superclass run method

    def test_onePlusOneEqualsTwo(self):
        self.assertTrue(1 + 1 == 2) # succeeds

    def test_onePlusOneEqualsThree(self):
        self.assertTrue(1 + 1 == 3) # fails

    def test_onePlusNoneIsNone(self):
        self.assertTrue(1 + None is None) # raises TypeError

if __name__ == '__main__':
    unittest.main()

EDIT: This works for Python 2.6 - 3.3, (modified for new Python bellow).

Answer 5

It depends what kind of reporting you'd like to produce.

In case you'd like to do some actions on failure (such as generating a screenshots), instead of using tearDown(), you may achieve that by overriding failureException.

For example:

@property
def failureException(self):
    class MyFailureException(AssertionError):
        def __init__(self_, *args, **kwargs):
            screenshot_dir = 'reports/screenshots'
            if not os.path.exists(screenshot_dir):
                os.makedirs(screenshot_dir)
            self.driver.save_screenshot('{0}/{1}.png'.format(screenshot_dir, self.id()))
            return super(MyFailureException, self_).__init__(*args, **kwargs)
    MyFailureException.__name__ = AssertionError.__name__
    return MyFailureException

Answer 6

This solution is for Python versions 2.7 to 3.7 (the highest current version), without any decorators or other modification in any code before tearDown. Everything works according to the builtin classification of results. Also skipped tests or expectedFailure are recognized correctly. It evaluates the result of the current test, not a summary of all tests passed so far. Compatible also with pytest.

import unittest

class MyTest(unittest.TestCase):
    def tearDown(self):
        if hasattr(self, '_outcome'):  # Python 3.4+
            result = self.defaultTestResult()  # these 2 methods have no side effects
            self._feedErrorsToResult(result, self._outcome.errors)
        else:  # Python 3.2 - 3.3 or 3.0 - 3.1 and 2.7
            result = getattr(self, '_outcomeForDoCleanups', self._resultForDoCleanups)
        error = self.list2reason(result.errors)
        failure = self.list2reason(result.failures)
        ok = not error and not failure

        # demo:   report short info immediately (not important)
        if not ok:
            typ, text = ('ERROR', error) if error else ('FAIL', failure)
            msg = [x for x in text.split('\n')[1:] if not x.startswith(' ')][0]
            print("\n%s: %s\n     %s" % (typ, self.id(), msg))

    def list2reason(self, exc_list):
        if exc_list and exc_list[-1][0] is self:
            return exc_list[-1][1]

    # DEMO tests
    def test_success(self):
        self.assertEqual(1, 1)

    def test_fail(self):
        self.assertEqual(2, 1)

    def test_error(self):
        self.assertEqual(1 / 0, 1)

Comments: Only one or zero exceptions (error or failure) need be reported because not more can be expected before tearDown. The package unittest expects that a second exception can be raised by tearDown. Therefore the lists errors and failures can contain only one or zero elements together before tearDown. Lines after "demo" comment are reporting a short result.

Demo output: (not important)

$ python3.5 -m unittest test

EF.
ERROR: test.MyTest.test_error
     ZeroDivisionError: division by zero
FAIL: test.MyTest.test_fail
     AssertionError: 2 != 1

==========================================================
... skipped usual output from unittest with tracebacks ...
...
Ran 3 tests in 0.002s

FAILED (failures=1, errors=1)

---

Comparision to other solutions - (with respect to commit history of Python source repository):

• This solution uses a private attribute of TestCase instance like many other solutions, but I checked carefully all relevant commits in the Python source repository that three alternative names cover the code history since Python 2.7 to 3.6.2 without any gap. It can be a problem after some new major Python release, but it could be clearly recognized, skipped and easily fixed later for a new Python. An advantage is that nothing is modified before running tearDown, it should never break the test and all functionality of unittest is supported, works with pytest and it could work many extending packages, but not with nosetest (not a suprise becase nosetest is not compatible e.g. with unittest.expectedFailure).

• The solutions with decorators on the user test methods or with a customized failureException (mgilson, Pavel Repin 2nd way, kenorb) are robust against future Python versions, but if everything should work completely, they would grow like a snow ball with more supported exceptions and more replicated internals of unittest. The decorated functions have less readable tracebacks (even more levels added by one decorator), they are more complicated for debugging and it is unpleassant if another more important decorator has a problem. (Thanks to mgilson the basic functionality is ready and known issues can be fixed.)

• The solution with modifired run method and catched result parameter

  • (scoffey) should work also for Python 2.6. The interpretation of results can be improved to requirements of the question, but nothing can work in Python 3.4+, because result is updated after tearDown call, never before.
  • Mark G.: (tested with Python 2.7, 3.2, 3.3, 3.4 and with nosetest)

• solution by exc_info() (Pavel Repin 2st way) works only with Python 2.

• Other solutions are principially similar, but less complete or with more disadvantages.

---

Explained by Python source repository
= Lib/unittest/case.py =
Python v 2.7 - 3.3

class TestCase(object):
    ...
    def run(self, result=None):
        ...
        self._outcomeForDoCleanups = result   # Python 3.2, 3.3
        # self._resultForDoCleanups = result  # Python 2.7
        #                                     # Python 2.6 - no result saved
        ...
        try:
            testMethod()
        except...   # many times for different exception classes
            result.add...(self, sys.exc_info())  # _addSkip, addError, addFailure
        ...
        try:
            self.tearDown()
        ...

Python v. 3.4 - 3.6

    def run(self, result=None):
        ...
        # outocome is a context manager to catch and collect different exceptions
        self._outcome = outcome  
        ...
        with outcome...(self):
            testMethod()
        ...
        with outcome...(self): 
            self.tearDown() 
        ... 
        self._feedErrorsToResult(result, outcome.errors)

Note (by reading Python commit messages): A reason why test results are so much decoupled from tests is memory leaks prevention. Every exception info can access to frames of the failed process state including all local variables. If a frame is assigned to a local variable in a code block that could also fail, then a cross memory refence could be easily created. It is not terrible, thanks to garbage collector, but the free memory can became fragmented more quickly than if the memory would be released correctly. This is a reason why exception information and traceback are converted very soon to strings and why temporary objects like self._outcome are encapsulated and are set to None in a finally block in order to memory leaks are prevented.