How to move files between two Amazon S3 Buckets us

问题:

I have to move files between one bucket to another with Python Boto API. (I need it to "Cut" the file from the first Bucket and "Paste" it in the second one). What is the best way to do that?

** Note: Is that matter if I have two different ACCESS KEYS and SECRET KEYS?

回答1:

I think the boto S3 documentation answers your question.

https://github.com/boto/boto/blob/develop/docs/source/s3_tut.rst

Moving files from one bucket to another via boto is effectively a copy of the keys from source to destination than removing the key from source.

You can get access to the buckets:

import boto

c = boto.connect_s3()
src = c.get_bucket('my_source_bucket')
dst = c.get_bucket('my_destination_bucket')

and iterate the keys:

for k in src.list():
    # copy stuff to your destination here
    dst.copy_key(k.key.name, src.name, k.key.name)
    # then delete the source key
    k.delete()

See also: Is it possible to copy all files from one S3 bucket to another with s3cmd?

回答2:

If you are using boto3 (the newer boto version) this is quite simple

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 'mybucket',
    'Key': 'mykey'
}
s3.meta.client.copy(copy_source, 'otherbucket', 'otherkey')

(Docs)

回答3:

awscli does the job 30 times faster for me than boto coping and deleting each key. Probably due to multithreading in awscli. If you still want to run it from your python script without calling shell commands from it, you may try something like this:

Install awscli python package:

sudo pip install awscli

And then it is as simple as this:

import os
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
    os.environ['LC_CTYPE'] = 'en_US.UTF-8'

from awscli.clidriver import create_clidriver
driver = create_clidriver()
driver.main('s3 mv source_bucket target_bucket --recursive'.split())

回答4:

Bucket name must be string not bucket object. Below change worked for me

for k in src.list():
    dst.copy_key(k.key, src.name, k.key)

回答5:

If you want to

Create a copy of an object that is already stored in Amazon S3.

then copy_object is the way to go in boto3.

How I do it:

import boto3

aws_access_key_id = ""
aws_secret_access_key = ""
bucket_from = ""
bucket_to = ""
s3 = boto3.resource(
    's3',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key
)
src = s3.Bucket(bucket_from)

def move_files():
    for archive in src.objects.all():
        # filters on archive.key might be applied here

        s3.meta.client.copy_object(
            ACL='public-read',
            Bucket=bucket_to,
            CopySource={'Bucket': bucket_from, 'Key': archive.key},
            Key=archive.key
        )

move_files()