How to fork a process in python/django?

Question

This is more of a Python general question however in a context of django.

For now I have this view in django which has to process a lot of data. Usually it takes the server (nginx with django running in proxy using) a couple of minutes to do it. Sometimes the server times out. I don't want to increase the time-out time in nginx. I realize that if I can fork a process in python in the django view so that the forked (child) process will do all the data crunching independently of the django view, then the view would be able to return the request to the user immediately (therefore never timing-out) and the child process would continue running in the background finishing up all the calculation.

So here is the question:

How can I fork an independent process in python (and if possible for the python code to be in the same file)? And if possible how can I assign a unix process priority level to it?

I looked at some of the ways of forking a process in python and it seems there are a few options. Which one is the best appropriate for this scenario?

Thank you.

Answer 1

If you really want to fork and set priority, you can use os.fork and os.nice, but I think the multiprocessing module or Celery would be more applicable here.

Answer 2

the 'best practice' answer is to use a queue manager, typically RabbitMQ or any backend handled by Django-celery.

Still, there are a few lighter options that do spawn a new thread. what these options usually lack is some way to track progress, or keep the number of threads under control.

check Django-utils to see if it's enough. if not, go for Celery.