Django Background Task

Question

I need to populate a SQLite database every few minutes in Django, but I want to serve stale data until the data is available for the database to be updated. (i.e. I don't want to block for the data to be gathered; the only time I can block is if there is a lock on the database, during which I have no choice.)

I also don't want to install a separate program or library.

How would I go about setting up another thread that could call save() on a bunch of models, without running into threading issues?

Answer 1

Depends on whether you need the update to look atomic from the point of view of the readers. If you don't mind seeing old and new data together, just create a custom management command that populates the data, and run it every few minutes from cron.

If you need it to look atomic, wrapping the all the writes in one SQLite transaction via django.db.transaction should probably provide you with the necessary locks.

Answer 2

Celery.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Celery is written in Python, but the protocol can be implemented in any language. It can also operate with other languages using webhooks.

Answer 3

Just a quick update on John Lehmann's answer: django-background-task was unmaintained and incompatible with newer Django version. We updated and extended it with new features a while ago and maintaining the new backward compatible package on Github. The new django-background-tasks app can be downloaded or installed from the PyPI.

Answer 4

Django Background Task is a databased-backed work queue for Django, loosely based around Ruby's DelayedJob library.

You decorate functions to create tasks:

@background(schedule=60)
def notify_user(user_id):
    # lookup user by id and send them a message
    user = User.objects.get(pk=user_id)
    user.email_user('Here is a notification', 'You have been notified')

Though you still need something which schedules those tasks. Some benefits include automatic retries for failed tasks, and setting maximum duration for a running task.

This does involves another dependency but could be useful to some readers without that restriction.

Answer 5

If you're looking for a lightweight solution for just executing stuff in background rather than a full-blown task management system, take a look at django-utils. It includes, among other things, an @async function decorator that will make a function execute asynchronously in a separate thread.

Use it like this:

from djutils.decorators import async

@async
def load_data_async():
    # this will be executed in a separate thread
    load_data()

Then you can call either the load_data_async function for background, or the normal load_data function for blocking execution.

Just make sure to install a version before 2.0, since that lacks the @async decorator.

Note: If even installing django-utils would be too much, you can simply download it and include the few required files in your project.

Answer 6

I had the same issue but didnt want to run a service like celery to solve the problem.

I found posix_spawn on linux systems. You can write manage.py commands that run in your full django environment. These commands can be executed in the background with this project.

If you need to pass data back to the website during the run, I use memcached.

https://github.com/lukedupin/django_posix_spawn