scrapy store images to amazon s3

问题:

I store images in my local server then upload to s3
Now I want to edit it to stored images directly to amazon s3

But ther is error:

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

here is my settings.py

AWS_ACCESS_KEY_ID = "XXXX"
AWS_SECRET_ACCESS_KEY = "XXXX"
IMAGES_STORE = 's3://how.are.you/'

Do I need to add something??

my scrapy edition: Scrapy==0.22.2

Please guide me,thank you!

回答1:

I found the cause of the problem is upload policy. The function Key.set_contents_from_string() takes argument policy, default set to S3FileStore.POLICY. So modify the code in scrapy/contrib/pipeline/files.py, change

 return threads.deferToThread(k.set_contents_from_string, buf.getvalue(),
                              headers=h, policy=self.POLICY)

to

 return threads.deferToThread(k.set_contents_from_string, buf.getvalue(),
                              headers=h)

Maybe you can try it, and share the result here.

回答2:

AWS_ACCESS_KEY_ID = "xxxxxx"
AWS_SECRET_ACCESS_KEY = "xxxxxx"
IMAGES_STORE = "s3://bucketname/virtual_path/"

how.are.you should be a S3 Bucket that exist into your S3 account, and it will store the images you upload. If you want to store images inside any virtual_path then you need to create this folder into your S3 Bucket.

回答3:

I think the problem is not in your code, actually the problem lies in permission, please check your credentials first and make sure your permissions to access and write on s3 bucket.

    import boto
    s3 = boto.connect_s3('access_key', 'secret_key')
    bucket = s3.lookup('bucket_name')
    key = bucket.new_key('testkey')
    key.set_contents_from_string('This is a test')
    key.delete()

If test run successfuly then look into your permission, for setting permission you can look at amazon configuration