gunicorn ERROR (abnormal termination)

Question

im running a fabric script that, amongst other things, is supposed to restart gunicorn on an ubuntu server, the command is below:

supervisorctl status projectname:gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP

the problem is, is that gunicorn doesnt appear to be running in the first place so the process cannot be killed, ive ssh'd into the amazon ec2 instance and ran

sudo supervisorctl restart projectname:gunicorn' 

and I get an error response that says:

projectname:gunicorn: ERROR (not running) projectname:gunicorn ERROR (abnormal termination)

so i attempted to start gunicorn by running

sudo supervisorctl start projectname:gunicorn

and the error says

'projectname:gunicorn: Error (abnormal termination)'

So I need gunicorn to run, and im having trouble acheiving this

Ive also checked the gunicorn log and the text below, below is the relevant output

2014-01-17 14:58:14 [12260] [INFO] Starting gunicorn 0.14.3
2014-01-17 14:58:14 [12260] [INFO] Listening at: http://127.0.0.1:9000 (12260)
2014-01-17 14:58:14 [12260] [INFO] Using worker: sync
2014-01-17 14:58:14 [12263] [INFO] Booting worker with pid: 12263
2014-01-17 14:58:14 [12264] [INFO] Booting worker with pid: 12264
2014-01-17 14:58:14 [12265] [INFO] Booting worker with pid: 12265
2014-01-17 14:58:14 [12266] [INFO] Booting worker with pid: 12266
2014-01-17 14:58:14 [12263] [INFO] Worker exiting (pid: 12263)
2014-01-17 14:58:14 [12264] [INFO] Worker exiting (pid: 12264)
2014-01-17 14:58:14 [12265] [INFO] Worker exiting (pid: 12265)
2014-01-17 14:58:14 [12266] [INFO] Worker exiting (pid: 12266)
Traceback (most recent call last):
File "/opt/screening/env/bin/gunicorn_django", line 9, in <module>
load_entry_point('gunicorn==0.14.3', 'console_scripts', 'gunicorn_django')()
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 129, in run
DjangoApplication("%prog [OPTIONS] [SETTINGS_PATH]").run()
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 129, in run
Arbiter(self).run()
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 184, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 279, in halt
self.stop()
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 327, in stop
self.reap_workers()
File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 413, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

also, here is the conf file

[program:gunicorn]
command=/opt/screening/env/bin/gunicorn_django --pythonpath . ce.settings -w 4 --bind             127.0.0.1:9000
directory=/opt/screening/repository
user=www-data
autostart=true
autorestart=true
stdout_logfile=/opt/screening/logs/gunicorn.log
redirect_stderr=true

[program:celeryd]
command=/opt/screening/env/bin/python manage.py celeryd --autoscale=16,2 -E -l INFO --pidfile=/opt/screening/tmp/pids/celeryd.pid
directory=/opt/screening/repository
user=www-data
autostart=true
autorestart=true
stdout_logfile=/opt/screening/logs/celeryd.log
redirect_stderr=true

[program:celerybeat]
command=/opt/screening/env/bin/python manage.py celerybeat -l INFO --    schedule=/opt/screening/tmp/celerybeat-schedule --    pidfile=/opt/screening/tmp/pids/celerybeat.pid
directory=/opt/screening/repository
user=www-data
autostart=true
autorestart=true
stdout_logfile=/opt/screening/logs/celerybeat.log
redirect_stderr=true

[program:celerycam]
command=/opt/screening/env/bin/python manage.py celerycam --    pidfile=/opt/screening/tmp/pids/celerycam.pid
directory=/opt/screening/repository
user=www-data
autostart=true
autorestart=true
stdout_logfile=/opt/screening/logs/celerycam.log
redirect_stderr=true

[group:screening]
programs=gunicorn,celeryd,celerybeat,celerycam

any ideas? I understand that this is a lot of text, any hints or pointers would be much appreciated

Thanks for reading,

edit:

ran unicorn on its own, activated the virtual env and ran

python manage.py run_gunicorn

the terminal printed the below output

2014-01-19 22:02:35 [14735] [INFO] Starting gunicorn 0.14.3
2014-01-19 22:02:35 [14735] [INFO] Listening at: http://127.0.0.1:8000 (14735)
2014-01-19 22:02:35 [14735] [INFO] Using worker: sync
2014-01-19 22:02:35 [14742] [INFO] Booting worker with pid: 14742

also ran the run server in the virtualenv:

python manage.py runserver 7000
Validating models...

0 errors found
Django version 1.3, using settings 'ce.settings'
Development server is running at http://127.0.0.1:7000/
Quit the server with CONTROL-C.

so no apparent errors there

edit 2:

have spoken to a couple other people about this, and was advised to look at the permissions for the gunicorn logs, here they are:

-rw-rw-r-- 1 www-data ubuntu    3270504 2014-01-19 23:23 gunicorn.log

the www-data user matches the one set in the supervisor config

edit 3: I ran the gunicorn command again, but this time added logging info:

gunicorn_django --pythonpath . ce.settings -w 4 --bind             127.0.0.1:9000 --debug --log-level debug

and received the following error message:

Traceback (most recent call last):
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 453, in spawn_worker
    worker.init_process()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 99, in init_process
    self.wsgi = self.app.wsgi()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 101, in wsgi
    self.callable = self.load()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 87, in load
mod = util.import_module("gunicorn.app.django_wsgi")
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 18, in <module>
    from django.core.management.validation import get_validation_errors
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/core/management/validation.py", line 3, in <module>
    from django.contrib.contenttypes.generic import GenericForeignKey, GenericRelation
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/contrib/contenttypes/generic.py", line 6, in <module>
    from django.db import connection
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/db/__init__.py", line 14, in <module>
    if not settings.DATABASES:
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/utils/functional.py", line 276, in __getattr__
    self._setup()
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/screening/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 89, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'ce.settings' (Is it on sys.path?): No module named ce.settings

2014-01-20 09:14:22 [31830] [INFO] Worker exiting (pid: 31830)
Traceback (most recent call last):
  File "/opt/screening/env/bin/gunicorn_django", line 9, in <module>
load_entry_point('gunicorn==0.14.3', 'console_scripts', 'gunicorn_django')()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 129, in run
    DjangoApplication("%prog [OPTIONS] [SETTINGS_PATH]").run()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 129, in run
    Arbiter(self).run()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 184, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 279, in halt
    self.stop()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 327, in stop
    self.reap_workers()
  File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 413, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

so it appears that the salient info is this:

ImportError: Could not import settings 'ce.settings' (Is it on sys.path?): No module named ce.settings

My settings are in a settings directory, and the init file is present, so the issue isnt that. Also the application starts on the runserver so the settings file must be importable

Answer 1

(The question was answered by the OP in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )

The OP wrote:

Solved the issue (I think)

as per the info in this link https://stackoverflow.com/a/19256794/2049067 , I added the project to the python path

export PYTHONPATH=:/my/path

then ran the gunicorn command again:

gunicorn_django --pythonpath . ce.settings -w 4 --bind             127.0.0.1:9000 --debug --log-level debug

and gunicorn is up and running, and the site is accessible, I exited the ssh and everything is (seemingly) still working. I should also add that before I set the pythonpath I changed the ownerwhip on the gunicorn log:

sudo chown -R www-data:www-data gunicorn.log

Though I dont know if that helped

& seeing how the application has been running for years I dont know how the project was removed from the pythonpath