Nginx upstream prematurely closed connection while

问题:

I am using nginx and node server to serve update requests. I get a gateway timeout when I request an update on large data. I saw this error from the nginx error logs :

2016/04/07 00:46:04 [error] 28599#0: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.77, server: gis.oneconcern.com, request: "GET /update_mbtiles/atlas19891018000415 HTTP/1.1", upstream: "http://127.0.0.1:7777/update_mbtiles/atlas19891018000415", host: "gis.oneconcern.com"

I googled for the error and tried everything I could, but I still get the error.

My nginx conf has these proxy settings:

    ##
    # Proxy settings
    ##

    proxy_connect_timeout 1000;
    proxy_send_timeout 1000;
    proxy_read_timeout 1000;
    send_timeout 1000;

This is how my server is configured

server {
listen 80;

server_name gis.oneconcern.com;
access_log /home/ubuntu/Tilelive-Server/logs/nginx_access.log;
error_log /home/ubuntu/Tilelive-Server/logs/nginx_error.log;

large_client_header_buffers 8 32k;
location / {
    proxy_pass http://127.0.0.1:7777;
    proxy_redirect off;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $http_host;
    proxy_cache_bypass $http_upgrade;
}

location /faults {
    proxy_pass http://127.0.0.1:8888;
    proxy_http_version 1.1;
    proxy_buffers 8 64k;
    proxy_buffer_size 128k;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

}

I am using a nodejs backend to serve the requests on an aws server. The gateway error shows up only when the update takes a long time (about 3-4 minutes). I do not get any error for smaller updates. Any help will be highly appreciated.

Node js code :

app.get("/update_mbtiles/:earthquake", function(req, res){
var earthquake = req.params.earthquake
var command = spawn(__dirname + '/update_mbtiles.sh', [ earthquake, pg_details ]);
//var output  = [];

command.stdout.on('data', function(chunk) {
//    logger.info(chunk.toString());
//     output.push(chunk.toString());
});

command.stderr.on('data', function(chunk) {
  //  logger.error(chunk.toString());
 //   output.push(chunk.toString());
});

command.on('close', function(code) {
    if (code === 0) {
        logger.info("updating mbtiles successful for " + earthquake);
        tilelive_reload_and_switch_source(earthquake);
        res.send("Completed updating!");
    }
    else {
        logger.error("Error occured while updating " + earthquake);
        res.status(500);
        res.send("Error occured while updating " + earthquake);
    }
});
});

function tilelive_reload_and_switch_source(earthquake_unique_id) {
tilelive.load('mbtiles:///'+__dirname+'/mbtiles/tipp_out_'+ earthquake_unique_id + '.mbtiles', function(err, source) {
    if (err) {
        logger.error(err.message);
        throw err;
    }
    sources.set(earthquake_unique_id, source); 
    logger.info('Updated source! New tiles!');
});
}

Thank you.

回答1:

I think that error from Nginx is indicating that the connection was closed by your nodejs server (i.e., "upstream"). How is nodejs configured?

回答2:

I solved this by setting a higher timeout value for the proxy:

location / {
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;
    proxy_pass http://localhost:3000;
}

Documentation: https://nginx.org/en/docs/http/ngx_http_proxy_module.html

回答3:

I had the same error for quite a while, and here what fixed it for me.

I simply declared in service that i use what follows:

Description= Your node service description
After=network.target

[Service]
Type=forking
PIDFile=/tmp/node_pid_name.pid
Restart=on-failure
KillSignal=SIGQUIT
WorkingDirectory=/path/to/node/app/root/directory
ExecStart=/path/to/node /path/to/server.js

[Install]
WantedBy=multi-user.target

What should catch your attention here is "After=network.target". I spent days and days looking for fixes on nginx side, while the problem was just that. To be sure, stop running the node service you have, launch the ExecStart command directly and try to reproduce the bug. If it doesn't pop, it just means that your service has a problem. At least this is how i found my answer.

For everybody else, good luck!

回答4:

You can increase the timeout in node like so.

app.post('/slow/request', function(req, res){ req.connection.setTimeout(100000); //100 seconds ... }

回答5:

I don't think this is your case, but I'll post it if it helps anyone. I had the same issue and the problem was that Node didn't respond at all (I had a condition that when failed didn't do anything - so no response) - So if increasing all your timeouts didn't solve it, make sure all scenarios get a response.

回答6:

In my case, I tried to increase the timeout in the configuration file, but did not work. Later turned out it was working when filtering for less data to display on one page. In the views.py, I just added " & Q(year=2019)" to only display the data for the year 2019. BTW, a permanent fix would be using Pagination.

def list_offers(request, list_type):
context = {}
context['list_type'] = list_type
if list_type == 'ready':
    context['menu_page'] = 'ready'
    offer_groups = OfferGroup.objects.filter(~Q(run_status=OfferGroup.DRAFT) & Q(year=2019)).order_by('-year', '-week')

context['grouped_offers'] = offer_groups

return render(request, 'app_offers/list_offers.html', context)