Running celeryd_multi with supervisor

Question

I'm working with djcelery and supervisor. I was running a celery with supervisor and everything worked fine, once I realized that I needed to change it to celery multi everything broke up. If I run celeryd_multi in a terminal it works but always run in background, like supervisor need that the command run in foreground there where the problem is.

This is my celery.ini:

[program:celery_{{ division }}]
command = {{ virtualenv_bin_dir }}/python manage.py celeryd_multi start default mailchimp -c:mailchimp 3 -c:default 5 --loglevel=info --logfile={{ log_dir }}/celery/%n.log --pidfile={{ run_dir }}/celery/%n.pid --schedule=/home/celery/celerybeat-schedule --settings={{ django_settings_python_path }}
autorestart = false
autostart = false
directory = {{ repo_dir }}/{{ division }}
user=celery
numprocs = 1
redirect_stderr = True
stopwaitsecs = 10
startsecs = 10
priority = 997
startretries = 3

Here is the command I put in the terminal that works fine

python manage.py celeryd_multi start default mailchimp -c:mailchimp 3 -c:default 5 --loglevel=info --logfile=/var/log/celery/%n.log --pidfile=/var/log/celery/%n.pid --schedule=/home/celery/celerybeat-schedule --settings=lively.settings_gunicorn

Answer 1

Daemontools has a utility called fghack designed to make a background process "stay" in the foreground.

pidsig supposedly is similar, but proxies signals:

http://permalink.gmane.org/gmane.comp.sysutils.supervision.general/2010

https://github.com/chexum/pidsig/blob/master/pidsig.c

I believe both are basically wrappers that wait on the backgrounded children.

Answer 2

Reply from the main developer of celery (23 Mar 2012):

I'm not aware of any simple solution to start multiple celeryd instances using supervisor, but you can always use one config for each. Maybe someone else has a solution for it though.

The generic-init.d/celeryd script uses celeryd-multi to start multiple servers, and you can use the CELERYD_NODES /etc/init.d/celeryd variable to set a number, or a list of worker names. I'm not sure how supervisord could be made to work with celeryd-multi, as it would probably need to know which of the resulting PIDs should be monitored or not (maybe a supervisord plugin could be written?)

-- Ask Solem