Multithreading for Python Django

问题:

Some functions should run asynchronously on the web server. Sending emails or data post-processing are typical use cases.

What is the best (or most pythonic) way write a decorator function to run a function asynchronously?

My setup is a common one: Python, Django, Gunicorn or Waitress, AWS EC2 standard Linux

For example, here's a start:

from threading import Thread

def postpone(function):
    def decorator(*args, **kwargs):
        t = Thread(target = function, args=args, kwargs=kwargs)
        t.daemon = True
        t.start()
    return decorator

desired usage:

@postpone
def foo():
    pass #do stuff

回答1:

I've continued using this implementation at scale and in production with no issues.

Decorator definition:

def start_new_thread(function):
    def decorator(*args, **kwargs):
        t = Thread(target = function, args=args, kwargs=kwargs)
        t.daemon = True
        t.start()
    return decorator

Example usage:

@start_new_thread
def foo():
  #do stuff

Over time, the stack has updated and transitioned without fail.

Originally Python 2.4.7, Django 1.4, Gunicorn 0.17.2, now Python 3.6, Django 2.1, Waitress 1.1.

If you are using any database transactions, Django will create a new connection and this needs to be manually closed:

from django.db import connection

@postpone
def foo():
  #do stuff
  connection.close()

回答2:

Celery is an asynchronous task queue/job queue. It's well documented and perfect for what you need. I suggest you start here

回答3:

The most common way to do asynchronous processing in Django is to use Celery and django-celery.