Running Celery as Daemon with Supervisor and Djang

Question

I am attempting to get Celery to run on an EB environment with Django and I have gotten super close to getting it all running. I have this config file for supervisor

#!/usr/bin/env bash

# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}

# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv,USER="ec2-user"

[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# Reread the supervisord config
echo reread
supervisorctl -c /opt/python/etc/supervisord.conf reread

echo reload
# supervisorctl -c /opt/python/etc/supervisord.conf reload

# Update supervisord in cache without restarting all services
echo update
supervisorctl -c /opt/python/etc/supervisord.conf update

# Start/Restart celeryd through supervisord
echo restart
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker

As it stands right now when I deploy it celery beat will run (verified by sshing in and using ps -aux | less to see the running processes) but no celeryd-worker. In my activity.log I have this.

[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-05-30T17:23:37.675Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

  directory=/opt/python/current/app
  user=ec2-user
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  reread
  No config updates to processes
  reload
  update
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)

so there is something wrong with when I try and start the worker but not with beat.

If I SSH into the EB and look at the /var/log/celery-worker.log file I see this.

[2017-05-30 12:30:21,370: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/hub.py", line 282, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 139, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 315, in _get_async
    q, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 412, in asynsqs
    AsyncSQSConnection, _asynsqs.regions(),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 400, in _aws_connect_to
    port=port)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/sqs/connection.py", line 39, in __init__
    https_connection_factory=https_connection_factory, **kwargs
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 223, in __init__
    AsyncConnection.__init__(self, http_client, **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 172, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

Which I also encountered when doing this on my local machine. Although I have made sure that curl is installed using nss with this container command.

container_commands:
  01_libcurl_ssl_backend:
    command: "export PYCURL_SSL_LIBRARY=nss"
    leader_only: true

Which is what pycurl is looking for. I also have pycurl in my requirements file and if I pip freeze I can see that pycurl is installed. I fixed this problem locally by recreating my virtual environment. But not quite sure what to do here.

Edit 1 I added in

PYCURL_SSL_LIBRARY: nss

to my options settings in my config files and was able to deploy and have a slightly different behaviour show up. When looking at the activity log after.

Instead of

update
restart
celeryd-beat: stopped
celeryd-beat: started
celeryd-worker: ERROR (not running)
celeryd-worker: ERROR (abnormal termination)

I got:

    reread
  celeryd-beat: changed
  celeryd-worker: changed
  reload
  update
  celeryd-beat: stopped
  celeryd-beat: updated process group
  celeryd-worker: stopped
  celeryd-worker: updated process group
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: stopped
  celeryd-worker: ERROR (abnormal termination)

but the celery-worker.log still has the same error in it about pycurl

Answer 1

1 - Follow the answer of this question: https://stackoverflow.com/a/41097851/3407986

2 - You must add libcurl-devel in packages eb configuration file/section, see my .ebextensions/02_packages.config file:

packages:
  yum:
    // ... your another packages where ...
    // add libcurl-devel, dependency of pycurl package
    libcurl-devel: []

Answer 2

Try SSH to your instance and importing pycurl yourself, using the same python that EB is using to run your app. Then you'll see the actual error.

For example, here's what I got when I built pycurl without --global-option="--with-nss":

[root@ip-172-31-2-149 log]# /opt/python/run/venv/local/bin/python Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. \>>> import pycurl Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)