Boto3 S3, sort bucket by last modified

问题:

I need to fetch a list of items from S3 using Boto3, but instead of returning default sort order (descending) I want it to return it via reverse order.

I know you can do it via awscli:

aws s3api list-objects --bucket mybucketfoo --query "reverse(sort_by(Contents,&LastModified))"

and its doable via the UI console (not sure if this is done client side or server side)

I cant seem to see how to do this in Boto3.

I am currently fetching all the files, and then sorting...but that seems overkill, especially if I only care about the 10 or so most recent files.

The filter system seems to only accept the Prefix for s3, nothing else.

回答1:

If there are not many objects in the bucket, you can use Python to sort it to your needs.

Define a lambda to get the last modified time:

get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s'))

Get all objects and sort them by last modified time.

s3 = boto3.client('s3')
objs = s3.list_objects_v2(Bucket='my_bucket')['Contents']
[obj['Key'] for obj in sorted(objs, key=get_last_modified)]

If you want to reverse the sort:

[obj['Key'] for obj in sorted(objs, key=get_last_modified, reverse=True)]

回答2:

I did a small variation of what @helloV posted below. its not 100% optimum, but it gets the job done with the limitations boto3 has as of this time.

s3 = boto3.resource('s3')
my_bucket = s3.Bucket('myBucket')
unsorted = []
for file in my_bucket.objects.filter():
   unsorted.append(file)

files = [obj.key for obj in sorted(unsorted, key=get_last_modified, 
    reverse=True)][0:9]

回答3:

it seems that is no way to do the sort by using boto3. According to the documentation, boto3 only supports these methods for Collections:

all(), filter(**kwargs), page_size(**kwargs), limit(**kwargs)

Hope this help in some way. https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.ServiceResource.buckets

回答4:

keys = []

kwargs = {'Bucket': 'my_bucket'}
while True:
    resp = s3.list_objects_v2(**kwargs)
    for obj in resp['Contents']:
        keys.append(obj['Key'])

    try:
        kwargs['ContinuationToken'] = resp['NextContinuationToken']
    except KeyError:
        break

this will get you all the keys in a sorted order

回答5:

A simpler approach, using the python3 sorted() function:

import boto3
s3 = boto3.resource('s3')

myBucket = s3.Bucket('name')

def obj_last_modified(myobj):
    return myobj.last_modified

sortedObjects = sorted(myBucket.objects.all(), key=obj_last_modified, reverse=True)

you now have a reverse sorted list, sorted by the 'last_modified' attribute of each Object.

回答6:

s3 = boto3.client('s3')

get_last_modified = lambda obj: int(obj['LastModified'].strftime('%Y%m%d%H%M%S'))

def sortFindLatest(bucket_name):
    resp = s3.list_objects(Bucket=bucket_name)
    if 'Contents' in resp:
        objs = resp['Contents']
        files = sorted(objs, key=get_last_modified)
        for key in files:
            file = key['Key']
            cx = s3.get_object(Bucket=bucket_name, Key=file)

This works for me to sort by date and time. I am using Python3 AWS lambda. Your mileage may vary. It can be optimized, I purposely made it discrete. As mentioned in an earlier post, 'reverse=True' can be added to change the sort order.