django-storages and amazon s3 - suspiciousoperatio

Question

I'm using django-storages with Amazon S3. I see the following error somewhat intermittently:

name = self._normalize_name(self._clean_name(name))\n\n  File \"/app/.heroku/venv/lib/python2.7/site-packages/storages/backends/s3boto.py\", line 237, in _normalize_name\n    name)\n\nSuspiciousOperation: Attempted access to 'https:/plantvillage.s3.amazonaws.com/avatar/hans9_avatar.jpg'

Note the single / after https:.

Does anyone know why this shows up? It doesn't happen all the time. I can successfully do this in other cases.

Answer 1

I fixed this, adding SuspiciousOperation on except:

class S3CustomStorage(S3BotoStorage):
    def _normalize_name(self, name):
        try:
            return safe_join(self.location, name)
        except (SuspiciousOperation, ValueError):
            return ""

Answer 2

When you use default_storage methods make sure to use the file.name:

Correct:

default_storage.delete(file.name)

Wrong:

default_storage.delete(file.url)

Wrong:

default_storage.delete(file)

All three examples above work with local files, but when using s3 you will run into this error unless you use file.name.

Answer 3

I haven't gotten S3 storage working on my own project yet, but I did just run across this error, and might be able to point you in the right direction.

If you look at S3BotoStorage._clean_name(), it's just: return os.path.normpath(name).replace('\\', '/'). os.path.normpath() converts the // in your URL to \\, and then .replace() converts that to \. Then, S3BotoStorage._normalize_name() checks to make sure this broken URL is part of the location it represents, which of course it's not. That's where the SuspiciousOperation error is being raised.

So 'name' looks like it's meant to be a local path, instead of the entire AWS URL. In my case, the immediate cause was FILEBROWSER_DIRECTORY = MEDIA_URL + "uploads/" in settings.py, which I had tried hoping to fix a different error about a missing upload folder.

Answer 4

_normalize_name does a lot of fancy and mostly unnecessary on Django stuff with the URL. In my case I just override the S3BotoStorage like this:

class S3CustomStorage(S3BotoStorage):
def _normalize_name(self, name):
    """
    Get rid of this crap: http://stackoverflow.com/questions/12535123/django-storages-and-amazon-s3-suspiciousoperation
    """
    return name

Then use it in the storage property:

ImageField(storage=S3CustomStorage())

And it worked for django simple ImageField with this base configuration:

AWS_ACCESS_KEY_ID = 'TTTT'
AWS_SECRET_ACCESS_KEY = 'XXXX'
AWS_STORAGE_BUCKET_NAME = 'ZZZZ'

Answer 5

Setting

MEDIA_ROOT=''

fixed the problem for me.