How to test coverage properly with Django + Nose

Question

Currently have a project configured to run coverage via Django's manage command like so:

./manage.py test --with-coverage --cover-package=notify --cover-branches --cover-inclusive --cover-erase

This results in a report like the following:

Name                        Stmts   Miss Branch BrMiss  Cover   Missing
--------------------------------------------------------------------------
notify.decorators               4      1      0      0    75%   4
notify.handlers                 6      1      2      0    88%   11
notify.notification_types      46     39      2      0    19%   8-55, 59, 62, 66
notify.notifications           51     51      0      0     0%   11-141
--------------------------------------------------------------------------
TOTAL                         107     92      4      0    17%   

There's a problem with this report however. It's wrong. Coverage is marking lines missing, despite the fact that they are indeed being covered by tests. For example, if I run the tests via nosetests instead of django's manage command I get the following correct report:

Name                        Stmts   Miss Branch BrMiss  Cover   Missing
-----------------------------------------------------------------------------
notify.decorators               4      0      0      0   100%   
notify.handlers                 6      0      2      0   100%   
notify.notification_types      46      0      2      0   100%   
notify.notifications           51     25      0      0    51%   13, 18, 23, 28, 33, 38, 43, 48, 53, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 116, 121, 126, 131, 136, 141
-----------------------------------------------------------------------------
TOTAL                         107     25      4      0    77%   

Google led me to the coverage website's FAQ, http://nedbatchelder.com/code/coverage/faq.html

Q: Why do the bodies of functions (or classes) show as executed, but the def lines do not?

This happens because coverage is started after the functions are defined. The definition lines are executed without coverage measurement, then coverage is started, then the function is called. This means the body is measured, but the definition of the function itself is not.

To fix this, start coverage earlier. If you use the command line to run your program with coverage, then your entire program will be monitored. If you are using the API, you need to call coverage.start() before importing the modules that define your functions.

The question is, can I run the coverage reports properly via Django's manage command? Or do I have to bypass manage to avoid the situation where coverage is started after the "missing" lines are executed?

Answer 1

I had the same problem using a remote interpreter in a virtual machine through the ssh configuration. The solution was to set my tests directory and ALL its parent directories in the "Path mappings" of the "Environment" section of "Run" > "Edit Configurations...".

Answer 2

I've managed to get this working including a

import coverage

on top of my manage.py file (I'm using Flask instead but having the very same issue)

My problem is that it works from console but Jenkins is not aware of it and keeps on saying that those imports are out of the tests...

Any idea?

Answer 3

As the docs say, "use the command line to run your program with coverage":

coverage run --branch --source=notify ./manage.py test

Answer 4

I spent sometime with this problem, and even with the answers given, they were not detailed enough to fully explain what I was experiencing. Here is what works well for me now, as per answer from iyn with a few necessary tweaks. My manage.py looks like this:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc

    # See https://stackoverflow.com/questions/24668174/how-to-test-coverage-properly-with-django-nose
    is_coverage_testing = 'test' in sys.argv and '--with-coverage' in sys.argv
    # Drop dupe with coverage arg
    if '--with-coverage' in sys.argv:
        sys.argv.remove('--with-coverage')

    if is_coverage_testing:
        import coverage
        cov = coverage.coverage(source=['client_app', 'config_app', 'list_app', 'core_app', 'feed_app',
                                        'content_app', 'lib',
                                        'job_app', 'license_app', 'search_app', 'weather_app'],
                                omit=['*/integration_tests/*'])
        cov.erase()
        cov.start()

    execute_from_command_line(sys.argv)

    if is_coverage_testing:
        cov.stop()
        cov.save()
        cov.report()

As can be seen above, I included all my apps for testing and excluded where I keep my integration tests.

My settings.py I dropped using cover package and with-coverage as this is already handled in manage.py now. Here is my settings with some explanations:

TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
# These are global options, trim as needed
# See https://stackoverflow.com/questions/24668174/how-to-test-coverage-properly-with-django-nose
NOSE_ARGS = [
    # '--cover-package=client_app',  # included in manage.py (hack to include all app testing)
    # '--cover-package=config_app',
    # '--cover-package=content_app',
    # '--cover-package=job_app',
    # '--cover-package=lib',
    # '--cover-package=license_app',
    # '--cover-package=list_app',
    # '--cover-package=search_app',
    # '--cover-package=core_app',
    # '--cover-package=weather_app',
    # '--cover-package=feed_app',
    '--logging-level=INFO',
    '--cover-erase',
    # '--with-coverage',  # Included in manage.py (hack), do not use here or will create multiple reports
    # '--cover-branches',  # Lowers coverage
    '--cover-html',  # generate HTML coverage report
    '--cover-min-percentage=59',
    # '--cover-inclusive',  # can't get coverage results on most files without this... This breaks django tests.
]

I run my basic tests like so (with coverage):

./manage.py test --noinput --verbose --with-coverage

And now I can see models.py, admins.py as well as apps.py gets covered.

I run my integration tests like so (without coverage):

./manage.py test integration_tests/itest_*  --noinput

I can also run a specific set of tests like so:

./manage.py test --noinput --verbose client_app/tests.py

You can also modify NOSE_ARGS as you wish or leave it out completely if you intend to use the flags each time on the command line. Cheers!

Answer 5

At the moment it's not possible to accurately run coverage alongside with django-nose (because of the way Django 1.7 loads models). So to get the coverage stats, you need to use coverage.py directly from command line, e.g:

$ coverage run --branch --source=app1,app2 ./manage.py test
$ coverage report
$ coverage html -d coverage-report

You can put coverage.py settings into .coveragerc file in the project root (the same dir as manage.py).

This issue is reported on django-nose GitHub page: https://github.com/django-nose/django-nose/issues/180 so maintainers know about the problem, you can let them know that you're also experiencing this issue.

UPDATE

eliangcs pointed out (django-nose issues on GiHub), that woraround is to modify your manage.py:

import os
import sys

if __name__ == "__main__":
    # ...
    from django.core.management import execute_from_command_line

    is_testing = 'test' in sys.argv

    if is_testing:
        import coverage
        cov = coverage.coverage(source=['package1', 'package2'], omit=['*/tests/*'])
        cov.erase()
        cov.start()

    execute_from_command_line(sys.argv)

    if is_testing:
        cov.stop()
        cov.save()
        cov.report()

It works, but it's rather "hacky" approach.

UPDATE 2

I recommend for everybody that uses nose to have a look at py.test (http://pytest.org/), which is really good Python testing tool, it integrates well with Django, has a lot of plugins and many more. I was using django-nose, but tried py.test and never looked back.