Django Celery Elastic Beanstalk supervisord no suc

问题:

My celery_config.txt in file script in .ebextensions

#!/usr/bin/env bash

# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}

# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv

[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread

# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update

# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

My celery-worker.log

[2017-04-18 18:48:06,042: CRITICAL/MainProcess] Unrecoverable error: ImportError('No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 143, in on_start
    self.emit_banner()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 158, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 221, in startup_info
    results=self.app.backend.as_uri(),
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 1182, in backend
    return self._get_backend()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 900, in _get_backend
    self.loader)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 65, in by_url
    return by_name(backend, loader), url
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 45, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com

My celery-beat.log

ImportError: No module named wellfie
Traceback (most recent call last):
  File "/opt/python/run/venv/bin/celery", line 9, in <module>
    load_entry_point('celery==4.0.2', 'console_scripts', 'celery')()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 503, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/utils.py", line 355, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 506, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named wellfie

MY eb-activity.log

[2017-04-18T19:50:23.354Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Starting activity...
[2017-04-18T19:50:28.095Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Completed activity. Result:
  httpd: stopped
  httpd: started
  httpd                            RUNNING   pid 11508, uptime 0:00:03
[2017-04-18T19:50:28.095Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"
  No config updates to processes
  celeryd: ERROR (no such process)
  celeryd: ERROR (no such process)
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/post.
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1] : Completed activity. Result:
  Application version switch - Command CMD-AppDeploy stage 1 completed
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Starting activity...
[2017-04-18T19:50:28.488Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.563Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Completed activity. Result:
  Disabled forced hourly log rotation.
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logpublish/hooks/config.
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Starting activity...
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.677Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Completed activity. Result:
  Log streaming option setting is not specified, ignore cloudwatch logs setup.

  Disabled log streaming.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logstreaming/hooks/config.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Completed activity.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45] : Completed activity. Result:
  Application update - Command CMD-AppDeploy succeeded
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...

I am not sure what's causing the error but this particular line caught my eye celeryd: ERROR (no such process). Locally my setup works like a charm, I get all the tasks, have zero issues running workers/beat, but on production it seems that supervisord is having difficulties accessing the celery worker, or that celery worker is not starting properly.

I have been debugging quite a bit, but I'm having issues figuring out what the problem is. I have removed the variables and urls, they all seem okay. If I ssh manually into the instance I can run both celery beat and celery worker without any trouble, but supervisord is having issues.

Also is this path correct?

; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

I am an AWS newbie, so I apologize if this solution is obvious.

EDIT:

01_packages.config

packages:
  yum:
    git: []
    postgresql93-devel: []
    libcurl-devel: []

02_python.config

container_commands:
  04_celery_tasks:
    command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true
  05_celery_tasks_run:
    command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true

django.config

option_settings:
  aws:elasticbeanstalk:application:environment:
     DJANGO_SETTINGS_MODULE: "wellfie.settings"
     PYTHONPATH: "/opt/python/current/app/wellfie:$PYTHONPATH"

  aws:elasticbeanstalk:container:python:
    WSGIPath: wellfie/wsgi.py

回答1:

Yes, the path is ok:

command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

I've notice that you have your PYTHONPATH variable set twice, like you would overwrite with eb environment property? Once it is set on application and then on application wellfie package.

Your log entry:

environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%ENV_PATH)s",
RDS_PORT="5432",
PYTHONPATH="/opt/python/current/app/wellfie:",
REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",
RDS_DB_NAME="bbbdb",
DJANGO_SETTINGS_MODULE="wellfie.settings",
RDS_USERNAME="aaaa",
RDS_PASSWORD="bbbb",
RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"

My:

environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",
WEBSERVER="APACHE",
CNAME="xxxxx-prod-env",
PYCURL_SSL_LIBRARY="nss",
APP_ENV="prod"

So I would start with investigating that. Another thing to have in mind is that if you use some AWS services like Redis, you need open connection on it with security group settings for that service, otherwise you will not be able to connect.

Things to have in mind:

1. If you play on the instance, remember about virtualenv. So you run you commands like:

   cd /opt/python/current/app && source /opt/python/run/venv/bin/activate && python manage.py

2. "ImportError: No module named 'wellfie'" may happen once, and be logged into celery-beat.log, celery-worker.log" since system is starting and process may not be ready yet.

3. ImportError('No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com',) looks like python initialization issue.

Modify file root_folder/django_app/__init__.py:

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from django_app.celery import app as celery_app

__all__ = ['celery_app']

Fix was to change:

supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd 

into

supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat