Django test parallel AppRegistryNotReady

Question

I am trying to understand how to run django tests in parallel with in memory sqlite3.

I have django app with that structure:

gbook
    order
        ...
        tests
            __init__.py
            test_a1.py
            test_b1.py
            utils.py

test_a1.py and test_b1.py contains same code:

import time
from order import models
from .utils import BackendTestCase


class ATestCase(BackendTestCase):
    def test_a(self):
        time.sleep(1)
        a = models.City.objects.count()
        self.assertEqual(a, a)

class BTestCase(BackendTestCase):
    def test_b(self):
        time.sleep(1)
        a = models.City.objects.count()
        self.assertEqual(a, a)

utils.py is:

from django.test import TestCase, Client
from order import models
from django.conf import settings

from order.utils import to_hash


class BackendTestCase(TestCase):
    fixtures = ['City.json', 'Agency.json']

    def setUp(self):
        self.client = Client()
        self.lang_codes = (i[0] for i in settings.LANGUAGES)
        ...

settings_test.py:

from .settings import *

DEBUG = False

TEMPLATE_DEBUG = False

STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'

PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher',]  # faster

DATABASES['default'] = {
    'ENGINE': 'django.db.backends.sqlite3',
}

When I run test in single process, all goes well (about 4 sec):

python.exe manage.py test order --settings=gbook.settings_test

Then I trying to run tests in parallel:

python.exe manage.py test order --settings=gbook.settings_test --parallel=2

I get this trace (console):

Creating test database for alias 'default'...

Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
    from order import models
  File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
    from order import models
  File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
    class Agency(models.Model):
  File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
    class Agency(models.Model):
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

From Pycharm trace is different:

...

Traceback (most recent call last):
  File "C:\python\Python36-32\lib\unittest\suite.py", line 163, in _handleClassSetUp
    setUpClass()
  File "C:\python\venv\gbook\lib\site-packages\django\test\testcases.py", line 1036, in setUpClass
    'database': db_name,
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "C:\python\venv\gbook\lib\site-packages\modeltranslation\management\commands\loaddata.py", line 61, in handle
    return super(Command, self).handle(*fixture_labels, **options)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 109, in loaddata
    self.load_label(fixture_label)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 175, in load_label
    obj.save(using=self.using)
  File "C:\python\venv\gbook\lib\site-packages\django\core\serializers\base.py", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 905, in _save_table
    forced_update)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\query.py", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 899, in execute_sql
    raise original_exception
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 889, in execute_sql
    cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "C:\python\venv\gbook\lib\site-packages\django\utils\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\sqlite3\base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: Problem installing fixture 'C:\kvk\develop\Python\gbook\order\fixtures\AirportInfo.json': Could not load order.AirportInfo(pk=2411): no such table: GB_AIRPORT_INFO

It seem like migrations not works for parallel, but why?

Docs says: "--parallel" Runs tests in separate parallel processes. Each process gets its own database. And I do not need to change my code for use it.

Please, help me to understand, what am i doing wrong.

multiprocessing.cpu_count() = 4

Django version 1.11.10

Python 3.6.5

Answer 1

I ran into a similar issue trying to use the --parallel feature on Windows.

Django's documentation states

This feature isn’t available on Windows. It doesn’t work with the Oracle database backend either.

Running the same command on Linux completed with no issues.