Django + S3 (boto) + Sorl Thumbnail: Suggestions f

Question

I am using S3 storage backend across a Django site I am developing, both to reduce load from the EC2 server(s), and to allow multiple webservers (redundancy, load balancing) access the same set of uploaded media.

Sorl.thumbnail (v11) template tags are being used in our templates to allow flexible image resizing/cropping.

Performance on media-rich pages is not very good, and when a page containing thumbnails needing to be generated for the first time is accessed, the requests even time out.

I understand that this is due to sorl thumbnail checking/downloading the original image from S3 (which could be quite large and high resolution), and rendering/checking/uploading the thumbnail.

What would you suggest is the best solution to this setup?

I have seen suggestions of storing a local copy of files in addition to the S3 copy (not to great when a couple of server are being used for load balancing). Also I've seen it suggested to store 0-byte files to fool sorl.thumbnail.

Are there any other suggestions or better ways of approaching this?

Answer 1

sorl thumbnail is now created with remote slow storages in mind. The first creation of the thumbnail is however done quering the storage, for example first accessed from template, but after that the references are cached in a key value store. Still you need the first query and creation, well one solution is to use the low level api sorl.thumbnail.get_thumbnail with the same options when the image is uploaded. When the image uploaded add this thumbnail creation job to a que like celery.

Answer 2

You can use Sorlery. It combines sorl and celery to create thumbnails via workers. It's very careful not to do any filesystem access outside of the worker thread.

The thumbnail returned immediately (before the worker has had a chance) can be controlled by setting your THUMBNAIL_DUMMY_SOURCE to an appropriate placeholder.

The job is created the first time the thumbnail is requested, subsequent requests are served the dummy image until the worker thread completes.

Answer 3

The easiest solution I've found so far is actually this third party service: http://cloudinary.com/

Answer 4

Just a heads-up for Flask users that need thumbnails on S3. I forked flask-thumbnails and created flask-thumbnails-s3. Unfortunately, doesn't do any fancy queueing or async stuff, like the various Django-based solutions linked above (contributions welcome!)... but it works.

Answer 5

Almost same as @Aidan's solution, I have made some tweaks on sorl-thumbnail. I also pre-generate thumbnails with celery. My code is here sorl_thumbnail-async

But I came to know easy_thumbnails does exactly what I was trying to do, so I am using it in my current project. You might find useful, short post on the topic is here