Django as S3 proxy

Question

I extended a ModelAdmin with a custom field "Download file", which is a link to a URL in my Django project, like:

http://www.myproject.com/downloads/1

There, I want to serve a file which is stored in a S3-bucket. The files in the bucket are not public readable, and the user may not have direct access to it. Now I want to

• avoid that the file has to be loaded in the server memory (these are multi-gb-files)
• avoid to have temp files on the server

The ideal solution would be to let django act as a proxy that streams S3-chunks directly to the user. I use boto, but did not find a possibility to stream the chunks. Any ideas?

Thanks.

Answer 1

Yes, you can use Django as a proxy to stream your S3 files to the user.

1. Generate a standard S3 url for your file.
2. Use the requests library to fetch your file in streaming mode.
3. Use Django to serve up the raw stream to the user.

import requests, boto3
from django.views.generic import View
from django.http.response import HttpResponse

class MediaDownloadView(View):

    def get(self, request, *args, **kwargs):
        s3 = boto3.resource(
            service_name='s3', aws_access_key_id='XXX_AWS_ACCESS_KEY_ID',
            aws_secret_access_key='XXX_AWS_ACCESS_KEY',
        )
        url = s3.meta.client.generate_presigned_url(
            ClientMethod="get_object", ExpiresIn=3600,
            Params={
                "Bucket": 'XXX_BUCKET_NAME',
                "Key": 'XXX_OBJECT_KEY',
            },
        )
        r = requests.get(url=url, stream=True)
        r.raise_for_status()
        response = HttpResponse(r.raw, content_type='audio/mp3')
        response['Content-Disposition'] = 'inline; filename=music.mp3'
        return response

Answer 2

Rather than proxying, why not simply redirect?

Use the django view at www.myproject.com/downloads/1 to serve a HTTP Redirect to the S3 storage URL - it's possible to generate time-limited authenticated URLs e.g. see here http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/

Then the client downloads the file directly from S3, but the contents is still secure and access has to come through your Django application