Problem with running Django with nginx and uwsgi i

Question

I'm trying to install a new server, and I can't run Django with nginx and uwsgi. I receive an error "502 Bad Gateway" and there are messages on the error log which I don't understand:

2019/07/20 10:50:44 [error] 2590#2590: *1 upstream prematurely closed connection while reading response header from upstream, client: 79.183.208.33, server: *.speedy.net.3.speedy-technologies.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/app/speedy_net/socket:", host: "3.speedy-technologies.com"

I have 4 websites and here is the main (default) configuration file:

server {
    listen [::]:80 default_server;
    listen 80 default_server;
    server_name *.speedy.net.3.speedy-technologies.com speedy.net.3.speedy-technologies.com;

    access_log /var/log/nginx/speedy-net.access.log;
    error_log /var/log/nginx/speedy-net.error.log;

    client_max_body_size 50M;

    root  /home/ubuntu/speedy-net/speedy/net/static_serve/root;
    try_files $uri @uwsgi;

    location @uwsgi {
        include uwsgi_params;
        uwsgi_pass unix:/run/uwsgi/app/speedy_net/socket;
    }

    location /admin/ {
        auth_basic "admin site";
        auth_basic_user_file /etc/nginx/htpasswd_admin;
        include uwsgi_params;
        uwsgi_pass unix:/run/uwsgi/app/speedy_net/socket;
    }

    location /static {
        alias /home/ubuntu/speedy-net/speedy/net/static_serve;
        access_log off;
        # expires max;
        gzip on;
        gzip_min_length 1100;
        gzip_buffers 4 32k;
        gzip_types text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
        gzip_vary on;
        gzip_comp_level 6;
    }
}

And:

[uwsgi]
project = net

chdir = /home/ubuntu/speedy-net
home = %(chdir)/env
module = speedy.%(project).wsgi:application

plugins = python3
master = true
processes = 4

chmod-socket = 666
vacuum = true

uid = ubuntu
gid = ubuntu

touch-reload = /run/uwsgi/app/speedy_%(project)/reload

I tried to test with sudo nginx -t but I received a message that my configuration is OK. What is the problem?

Edit:

The old server (which is working):

$ find /run/uwsgi/ -ls
  489      0 drwxr-xr-x   3 root     root           60 Jul 20 08:41 /run/uwsgi/
  490      0 drwxr-xr-x   6 root     root          120 Jul 20 08:41 /run/uwsgi/app
  500      0 drwxr-xr-x   2 ubuntu   ubuntu        100 Jul 20 08:42 /run/uwsgi/app/speedy_net
  523      0 -rw-rw-r--   1 ubuntu   ubuntu          0 Jul 20 08:43 /run/uwsgi/app/speedy_net/reload
  501      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 08:41 /run/uwsgi/app/speedy_net/socket
  457      4 -rw-r--r--   1 root     root            5 Jul 20 08:41 /run/uwsgi/app/speedy_net/pid
  498      0 drwxr-xr-x   2 ubuntu   ubuntu        100 Jul 20 08:42 /run/uwsgi/app/speedy_match
  524      0 -rw-rw-r--   1 ubuntu   ubuntu          0 Jul 20 08:43 /run/uwsgi/app/speedy_match/reload
  499      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 08:41 /run/uwsgi/app/speedy_match/socket
  413      4 -rw-r--r--   1 root     root            5 Jul 20 08:41 /run/uwsgi/app/speedy_match/pid
  495      0 drwxr-xr-x   2 ubuntu   ubuntu        100 Jul 20 08:43 /run/uwsgi/app/speedy_mail
  528      0 -rw-rw-r--   1 ubuntu   ubuntu          0 Jul 20 08:43 /run/uwsgi/app/speedy_mail/reload
  497      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 08:41 /run/uwsgi/app/speedy_mail/socket
  496      4 -rw-r--r--   1 root     root            5 Jul 20 08:41 /run/uwsgi/app/speedy_mail/pid
  491      0 drwxr-xr-x   2 ubuntu   ubuntu        100 Jul 20 08:42 /run/uwsgi/app/speedy_composer
  527      0 -rw-rw-r--   1 ubuntu   ubuntu          0 Jul 20 08:43 /run/uwsgi/app/speedy_composer/reload
  494      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 08:41 /run/uwsgi/app/speedy_composer/socket
  493      4 -rw-r--r--   1 root     root            5 Jul 20 08:41 /run/uwsgi/app/speedy_composer/pid

The new server:

$ find /run/uwsgi/ -ls
  554      0 drwxr-xr-x   3 root     root           60 Jul 20 10:15 /run/uwsgi/
  555      0 drwxr-xr-x   6 root     root          120 Jul 20 11:14 /run/uwsgi/app
  573      0 drwxr-xr-x   2 ubuntu   ubuntu         80 Jul 20 11:14 /run/uwsgi/app/speedy_net
  598      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 11:14 /run/uwsgi/app/speedy_net/socket
  584      4 -rw-r--r--   1 root     root            5 Jul 20 11:14 /run/uwsgi/app/speedy_net/pid
  568      0 drwxr-xr-x   2 ubuntu   ubuntu         80 Jul 20 11:14 /run/uwsgi/app/speedy_match
  572      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 11:14 /run/uwsgi/app/speedy_match/socket
  569      4 -rw-r--r--   1 root     root            5 Jul 20 11:14 /run/uwsgi/app/speedy_match/pid
  565      0 drwxr-xr-x   2 ubuntu   ubuntu         80 Jul 20 11:14 /run/uwsgi/app/speedy_mail
  567      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 11:14 /run/uwsgi/app/speedy_mail/socket
  566      4 -rw-r--r--   1 root     root            5 Jul 20 11:14 /run/uwsgi/app/speedy_mail/pid
  499      0 drwxr-xr-x   2 ubuntu   ubuntu         80 Jul 20 11:14 /run/uwsgi/app/speedy_composer
  564      0 srw-rw-rw-   1 ubuntu   ubuntu          0 Jul 20 11:14 /run/uwsgi/app/speedy_composer/socket
  561      4 -rw-r--r--   1 root     root            5 Jul 20 11:14 /run/uwsgi/app/speedy_composer/pid

The old server is Ubuntu 16.04.6 LTS, and the new server is Ubuntu 18.04.2 LTS.

I checked /var/log/uwsgi/app/speedy_net.log and it contains lots of lines. Here are the last 5 lines:

/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7fe09b2ac393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7fe09b277b70]
*** end of backtrace ***
Sun Aug 11 11:24:35 2019 - DAMN ! worker 2 (pid: 6874) died :( trying respawn ...
Sun Aug 11 11:24:35 2019 - Respawned uWSGI worker 2 (new pid: 6906)

The last 7 lines are now:

/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13a) [0x7f4d7c245e2a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7f4d7c31a393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
*** end of backtrace ***
Sun Aug 11 11:37:17 2019 - DAMN ! worker 2 (pid: 7915) died :( trying respawn ...
Sun Aug 11 11:37:17 2019 - Respawned uWSGI worker 2 (new pid: 7944)
Sun Aug 11 11:37:17 2019 - epoll_ctl(): File exists [core/event.c line 524]

And:

Sun Aug 11 11:42:56 2019 - epoll_ctl(): File exists [core/event.c line 524]
Sun Aug 11 11:42:56 2019 - epoll_ctl(): File exists [core/event.c line 524]
Sun Aug 11 11:42:56 2019 - !!! uWSGI process 8118 got Segmentation Fault !!!
Sun Aug 11 11:42:56 2019 - *** backtrace of 8118 ***
/usr/bin/uwsgi(uwsgi_backtrace+0x2a) [0x563d84f5098a]
/usr/bin/uwsgi(uwsgi_segfault+0x23) [0x563d84f50d73]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f4d7ff98f20]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_cleanup_key_block+0xb) [0x7f4d773d4cab]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_clear+0x16) [0x7f4d773d25a6]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(tls1_clear+0x9) [0x7f4d773de219]
/usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_new+0x3f2) [0x7f4d80c11a22]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x23969) [0x7f4d77669969]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x24fc5) [0x7f4d7766afc5]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectPoll+0xb78) [0x7f4d77654ba8]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0xfa28) [0x7f4d77655a28]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectdb+0x1f) [0x7f4d7765841f]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x12b01) [0x7f4d778a9b01]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x1388f) [0x7f4d778aa88f]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x122342) [0x7f4d7c220342]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_CallFunction_SizeT+0x124) [0x7f4d7c334ce4]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0xc258) [0x7f4d778a3258]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyCFunction_Call+0x96) [0x7f4d7c2460a6]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x7a50) [0x7f4d7c2e8450]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x3e) [0x7f4d7c31a67e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1beeb3) [0x7f4d7c2bceb3]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyObject_Call+0x48) [0x7f4d7c334b08]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x4001) [0x7f4d7c2e4a01]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c29e) [0x7f4d7c31a29e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c29e) [0x7f4d7c31a29e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ea5) [0x7f4d7c2e48a5]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1c85ec) [0x7f4d7c2c65ec]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1bc496) [0x7f4d7c2ba496]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1bc970) [0x7f4d7c2ba970]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x122342) [0x7f4d7c220342]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c058) [0x7f4d7c31a058]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x2c3) [0x7f4d7c319fb3]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x131) [0x7f4d7c333ec1]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcd) [0x7f4d7c334a9d]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x270821) [0x7f4d7c36e821]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0xb8ef5) [0x7f4d7c1b6ef5]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1ec989) [0x7f4d7c2ea989]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13a) [0x7f4d7c245e2a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7f4d7c31a393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
*** end of backtrace ***
Sun Aug 11 11:42:57 2019 - DAMN ! worker 2 (pid: 8118) died :( trying respawn ...
Sun Aug 11 11:42:57 2019 - Respawned uWSGI worker 2 (new pid: 8124)
Sun Aug 11 11:42:57 2019 - epoll_ctl(): File exists [core/event.c line 524]

The connection to the database (PostgreSQL) works from the command line, for example this code works fine:

import django

django.setup()

from django.contrib.sites.models import Site

for site in Site.objects.all().order_by('pk'):
    print(site.id, site.domain)

Answer 1

There are multiple issues that may be there, you need to debug and see what is wrong

1. ALLOWED_HOSTS is not set

If you the django config doesn't have ALLOWED_HOSTS set then this may create an error

2. nginx config is missing some directives

Some people were able to fix their issues using below

sendfile        on;
client_max_body_size 20M;
keepalive_timeout  0;
large_client_header_buffers 8 32k;

See if adding that to your route helps

3. Django app is crashing with some unhandled exception

This requires you to look at the uwsgi logs. You can do that by running the uwsgi process in one terminal instead of a as a service or set the log path in uwsgi.ini

[uwsgi]
logto=/var/log/uwsgi.log
log-ioerror=1

And then look at logs to see what is causing the crash

4. Your request is taking time and getting timed out

In this case you need to increase the uwsgi timeout. This you can do by adding below to uwsgi.ini

[uwsgi]
harakiri = 120

and below to the nginx config

uwsgi_read_timeout 120s;
uwsgi_send_timeout 120s;

5. uwsgi crash with segmentation fault

If you look at the logs you shared

/usr/bin/uwsgi(uwsgi_segfault+0x23) [0x563d84f50d73]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f4d7ff98f20]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_cleanup_key_block+0xb) [0x7f4d773d4cab]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_clear+0x16) [0x7f4d773d25a6]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(tls1_clear+0x9) [0x7f4d773de219]
/usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_new+0x3f2) [0x7f4d80c11a22]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x23969) [0x7f4d77669969]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x24fc5) [0x7f4d7766afc5]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectPoll+0xb78) [0x7f4d77654ba8]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0xfa28) [0x7f4d77655a28]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectdb+0x1f) [0x7f4d7765841f]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x12b01) [0x7f4d778a9b01]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x1388f) [0x7f4d778aa88f]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x122342) [0x7f4d7c220342]

These show that the crash happens because of the Postgres DB connection and ssl. Now there are lot of threads which talk about different solutions that can help you resolve the issue

uwsgi segmentation fault when serving a django application

django app under uwsgi failing to start when using postgresql database

Error importing psycopg2._psycopg when using uWSGI

uwsgi segmentation fault for one specific route

References

https://serverfault.com/questions/276462/nginx-uwsgi-gives-502-bad-gateway

"upstream prematurely closed connection while reading response header from upstream" Django, Ubuntu, Nginx, Gunicorn

uWSGI request timeout in Python

upstream prematurely closed connection (uwsgi + nginx + django)

nginx uwsgi websockets 502 Bad Gateway upstream prematurely closed connection while reading response header from upstream

uwsgi + nginx + flask: upstream prematurely closed

nginx, uwsgi, DJango, 502 when DEBUG=False, "upstream prematurely closed connection"

Answer 2

Eventually it was https://stackoverflow.com/a/54253374/57952. I had to run the following 2 commands:

/home/ubuntu/speedy-net/env/bin/pip uninstall psycopg2
/home/ubuntu/speedy-net/env/bin/pip install --no-binary :all: psycopg2==2.7.7

And then the server started to work.

Answer 3

Try adding a socket directive to the uwsgi config file.

socket = /run/uwsgi/app/speedy_%(project)/socket

You have to tell uwsgi where to find the socket. Maybe on the old server the socket path is passed as a command line flag?

Both the nginx user and the uswgi user should have rw permissions to the socket itself and rx permissions to every parent directory. You can check that both users can write to the socket by running touch as the corresponding users.

sudo --user=nginx touch /run/uwsgi/app/speedy_net/socket
sudo --user=uwsgi touch /run/uwsgi/app/speedy_net/socket

If any of these fail, there's a permission problem.