saving an image to bytes and uploading to boto3 re

Question

I'm trying to pull an image from s3, quantize it/manipulate it, and then store it back into s3 without saving anything to disk (entirely in-memory). I was able to do it once, but upon returning to the code and trying it again it did not work. The code is as follows:

import boto3
import io
from PIL import Image

client = boto3.client('s3',aws_access_key_id='',
        aws_secret_access_key='')
cur_image = client.get_object(Bucket='mybucket',Key='2016-03-19 19.15.40.jpg')['Body'].read()

loaded_image = Image.open(io.BytesIO(cur_image))
quantized_image = loaded_image.quantize(colors=50)
saved_quantized_image = io.BytesIO()
quantized_image.save(saved_quantized_image,'PNG')
client.put_object(ACL='public-read',Body=saved_quantized_image,Key='testimage.png',Bucket='mybucket')

The error I received is:

botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation: The Content-MD5 you specified did not match what we received.

It works fine if I just pull an image, and then put it right back without manipulating it. I'm not quite sure what's going on here.

Answer 1

The file may need to be saved and reloaded before you send it off to S3. The file pointer seek also needs to be at 0.

My problem was sending a file after reading out the first few bytes of it. Opening a file cleanly did the trick.

Answer 2

I found this question getting the same error trying to upload files -- two scripts clashed, one creating, the other uploading. My answer was to create using ".filename" then:

os.rename(filename.replace(".filename","filename"))

The upload script then needs to ignore . files. This ensured the file was done being created.

Answer 3

I had this same problem, and the solution was to seek to the beginning of the saved in-memory file:

out_img = BytesIO()
image.save(out_img, img_type)
out_img.seek(0)  # Without this line it fails
self.bucket.put_object(Bucket=self.bucket_name,
                       Key=key,
                       Body=out_img)