Can I have some code constantly run inside Django

Question

I'm using mod_wsgi to serve a django site through Apache. I also have some Python code that runs as a background process (dameon?). It keeps polling a server and inserts data into one of the Django models. This works fine but can I have this code be a part of my Django application and yet able to constantly run in the background? It doesn't need to be a process per se but a art of the Django site that is active constantly. If so, could you point me to an example or some documentation that would help me accomplish this?

Thanks.

Answer 1

You could either set up a cron job that runs some function you have defined, or - the more advanced and probably recommended method, integrate celery in your project (which is quite easy, actually).

Answer 2

You could create a background thread from the WSGI script when it is first being imported.

import threading
import time

def do_stuff():
    time.sleep(60)
    ... do periodic job

_thread = threading.Thread(target=do_stuff)
_thread.setDaemon(True)
_thread.start()

For this to work though you would have to be using only one daemon process otherwise each process would be doing the same thing which you probably do not want.

If you are using multiple process in daemon process group, an alternative is to create a special daemon process group which the only purpose of is to run this background thread. In other words, the process doesn't actually receive any requests.

You can do this by having:

WSGIDaemonProcess django-jobs processes=1 threads=1
WSGIImportScript /usr/local/django/mysite/apache/django.wsgi \
    process-group=django-jobs application-group=%{GLOBAL}

The WSGIImportScript directive says to load that script and run it on startup in the context of the process group 'django-jobs'.

To save having multiple scripts, I have pointed it at what would be your original WSGI script file you used for WSGIScriptAlias. We don't want it to run when it is loaded by that directive though, so we do:

import mod_wsgi

if mod_wsgi.process_group == 'django-jobs':
    _thread = threading.Thread(target=do_stuff)
    _thread.setDaemon(True)
    _thread.start()

Here it looks at the name of the daemon process group and only runs when started up within the special daemon process group set up with single process just for this.

Overall you are just using Apache as a big gloried process manager, albeit one which is already known to be robust. It is a bit of overkill as this process will consume additional memory on top of those accepting and handling requests, but depending on the complexity of what you are doing it can still be useful.

One cute aspect of doing this is that since it is still a full Django application in there, you could map specific URLs to just this process and so provide a remote API to manage or monitor the background task and what it is doing.

WSGIDaemonProcess django-jobs processes=1 threads=1
WSGIImportScript /usr/local/django/mysite/apache/django.wsgi \
    process-group=django-jobs application-group=%{GLOBAL}

WSGIDaemonProcess django-site processes=4 threads=5
WSGIScriptAlias / /usr/local/django/mysite/apache/django.wsgi

WSGIProcessGroup django-site
WSGIApplicationGroup %{GLOBAL}

<Location /admin>
WSGIProcessGroup django-jobs
</Location>

Here, all URLs except for stuff under /admin run in 'django-site', with /admin in 'django-jobs'.

Anyway, that addresses the specific question of doing it within the Apache mod_wsgi daemon process as requested.

As pointed out, the alternative is to have a command line script which sets up and loads Django and does the work and execute that from a cron job. A command line script means occasional transient memory usage, but startup cost for job is higher as need to load everything each time.

Answer 3

I previously used a cron job but I telling you, you will switch to celery after a while.

Celery is the way to go. Plus you can tasked long async process so you can speed up the request/response time.