NGinx & Django, serving large files (3gb+)

Question

I'm having some problems to serve large file downloads/uploads (3gb+).

As I'm using Django I guess that the problem to serve the file can become from Django or NGinx.

In my NGinx enabled site I have

server {
    ...
    client_max_body_size 4G;
    ...
}

And at django I'm serving the files in chunk sizes:

def return_file(path):
        filename = os.path.basename(path)
        chunk_size = 8192
        response = StreamingHttpResponse(FileWrapper(open(path), chunk_size), content_type=mimetypes.guess_type(path)[0])
        response['Content-Length'] = os.path.getsize(path)    
        response['Content-Disposition'] = 'attachment; filename={0}'.format(filename)
        return response

This method allowed me to pass from downloads of 600Mb~ to 2.6Gb, but it seems that the downloads are getting truncated at 2.6Gb. I traced the error:

2015/09/04 11:31:30 [error] 831#0: *553 upstream prematurely closed connection while reading upstream, client: 127.0.0.1, server: localhost, request: "GET /chat/download/photorec.zip/ HTTP/1.1", upstream: "http://unix:/web/rsmweb/run/gunicorn.sock:/chat/download/photorec.zip/", host: "localhost", referrer: "http://localhost/chat/2/" 

After reading some posts I added the following to my NGinx conf:

   proxy_read_timeout 300;
   proxy_connect_timeout 300;
   proxy_redirect off;

But I got the same error with an *1 instead of a *553*

I also thought that It could be a Django database Timeout, so I added:

DATABASE_OPTIONS = {
    'connect_timeout': 14400,
}

But it is not working either. (the download over the development server takes about 30 seconds)

PS: Some one already pointed me that the problem is Django, but I haven't been able to figure out why. Django is not printing or loggin any error!

Thanks for any help!

Answer 1

I ran into a similar problem which was visible in the nginx error log files by lines like this:

<TIMESTAMP> [error] 1221#1221: *913310 upstream prematurely closed connection 
while reading upstream, client: <IP>, server: <IP>, request: "GET <URL> HTTP/1.1",
upstream: "http://unix:<LOCAL_DJANGO_APP_DIR_PATH>/run/gunicorn.sock:
<REL_PATH_LOCAL_FILE_TO_BE_DOWNLOADED>", host: "<URL>", referrer: "<URL>/<PAGE>"

This is caused by the --timeout setting in the file

<LOCAL_DJANGO_APP_DIR_PATH>/bin/gunicorn_start 
(found at "command:" in /etc/supervisor/conf.d/<APPNAME>.conf)

In the gunicorn_start file change this line:

exec /usr/local/bin/gunicorn [...] \
--timeout <OLD_TIMEOUT> \
[...]

This was set to 300 and I had to change it to 1280 (this is in seconds!). Transfers of ~5GB are easily handled this way without RAM issues using

django.views.static.serve(request, <LOCAL_FILE_NAME>, <LOCAL_FILE_DIR>

Answer 2

Don't use django to deliver static content, specially not when it's static content that's as large as this. Nginx is ideal for delivering them. All you need to do is to create a mapping such as this in your nginx configuration file:

location /static/ {
     try_files $uri =404 ;
     root /var/www/myapp/;
     gzip on;
     gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

}

With /var/www/myapp/ being the top level folder for your django app. Inside that you will have a folder named static/ into which you need to collect all your static files with the django manage.py 's collectstatic command.

Of course you are free to rename these folders anyway you like and to use a different file structure all together. More about how to configure nginx for static content at this link: http://nginx.org/en/docs/beginners_guide.html#static