Celery: Worker with concurrency and reserved tasks

Question

Some of the tasks in my code were taking longer and longer to execute.

Upon inspection I noticed that although I have my worker node set to concurrency 6, and 6 processes exist to 'do work', but only 1 task is shown under 'running tasks'. Here is a little visual proof:

Here are the worker options:

And here is the task tab for that worker with only 1 running process:

I have found that if I restart celery, the concurrency is once again respected and i will see >1 running task, but after some amount of time/tasks it reverts back to this behavior..

Any ideas for fixing this intermittent problem?

Answer 1

I'm not sure if it's your use case, but I ran into similar problems when I had a mix of long and short tasks. Basically what happened is that at some point a process could start a very long running task, while prefetching a few other tasks, preventing them from being consumed by other processes. So I disabled the prefetching stuff, which is useful only if you're running a lot of short tasks.

To disable the prefetch, you need Celery 3.1+ and the Ofair setting, for instance:

celery -A proj worker -l info -Ofair