How do you search an amazon s3 bucket?

问题:

I have a bucket with thousands of files in it. How can I search the bucket? Is there a tool you can recommend?

回答1:

S3 doesn't have a native "search this bucket" since the actual content is unknown - also, since S3 is key/value based there is no native way to access many nodes at once ala more traditional datastores that offer a (SELECT * FROM ... WHERE ...) (in a SQL model).

What you will need to do is perform ListBucket to get a listing of objects in the bucket and then iterate over every item performing a custom operation that you implement - which is your searching.

回答2:

Just a note to add on here: it's now 3 years later, yet this post is top in Google when you type in "How to search an S3 Bucket."

Perhaps you're looking for something more complex, but if you landed here trying to figure out how to simply find an object (file) by it's title, it's crazy simple:

open the bucket, select "none" on the right hand side, and start typing in the file name.

http://docs.aws.amazon.com/AmazonS3/latest/UG/ListingObjectsinaBucket.html

回答3:

Here's a short and ugly way to do search file names using the AWS CLI:

aws s3 ls s3://your-bucket --recursive | grep your-search | cut -c 32-

回答4:

There are (at least) two different use cases which could be described as "search the bucket":

1. Search for something inside every object stored at the bucket; this assumes a common format for all the objects in that bucket (say, text files), etc etc. For something like this, you're forced to do what Cody Caughlan just answered. The AWS S3 docs has example code showing how to do this with the AWS SDK for Java: Listing Keys Using the AWS SDK for Java (there you'll also find PHP and C# examples).

2. List item Search for something in the object keys contained in that bucket; S3 does have partial support for this, in the form of allowing prefix exact matches + collapsing matches after a delimiter. This is explained in more detail at the AWS S3 Developer Guide. This allows, for example, to implement "folders" through using as object keys something like

   folder/subfolder/file.txt

   If you follow this convention, most of the S3 GUIs (such as the AWS Console) will show you a folder view of your bucket.

回答5:

There are multiple options, none being simple "one shot" full text solution:

1. Key name pattern search: Searching for keys starting with some string- if you design key names carefully, then you may have rather quick solution.

2. Search metadata attached to keys: when posting a file to AWS S3, you may process the content, extract some meta information and attach this meta information in form of custom headers into the key. This allows you to fetch key names and headers without need to fetch complete content. The search has to be done sequentialy, there is no "sql like" search option for this. With large files this could save a lot of network traffic and time.

3. Store metadata on SimpleDB: as previous point, but with storing the metadata on SimpleDB. Here you have sql like select statements. In case of large data sets you may hit SimpleDB limits, which can be overcome (partition metadata across multiple SimpleDB domains), but if you go really far, you may need to use another metedata type of database.

4. Sequential full text search of the content - processing all the keys one by one. Very slow, if you have too many keys to process.

We are storing 1440 versions of a file a day (one per minute) for couple of years, using versioned bucket, it is easily possible. But getting some older version takes time, as one has to sequentially go version by version. Sometime I use simple CSV index with records, showing publication time plus version id, having this, I could jump to older version rather quickly.

As you see, AWS S3 is not on it's own designed for full text searches, it is simple storage service.

回答6:

AWS released a new Service to query S3 buckets with SQL: Amazon Athena https://aws.amazon.com/athena/

回答7:

Given that you are in AWS...I would think you would want to use their CloudSearch tools. Put the data you want to search in their service...have it point to the S3 keys.

http://aws.amazon.com/cloudsearch/

回答8:

Search by Prefix in S3 Console

directly in the AWS Console bucket view.

Copy wanted files using s3-dist-cp

When you have thousands or millions of files another way to get the wanted files is to copy them to another location using distributed copy. You run this on EMR in a Hadoop Job. The cool thing about AWS is that they provide their custom S3 version s3-dist-cp. It allows you to group wanted files using a regular expression in the groupBy field. You can use this for example in a custom step on EMR

[
    {
        "ActionOnFailure": "CONTINUE",
        "Args": [
            "s3-dist-cp",
            "--s3Endpoint=s3.amazonaws.com",
            "--src=s3://mybucket/",
            "--dest=s3://mytarget-bucket/",
            "--groupBy=MY_PATTERN",
            "--targetSize=1000"
        ],
        "Jar": "command-runner.jar",
        "Name": "S3DistCp Step Aggregate Results",
        "Type": "CUSTOM_JAR"
    }
]

回答9:

Another option is to mirror the S3 bucket on your web server and traverse locally. The trick is that the local files are empty and only used as a skeleton. Alternatively, the local files could hold useful meta data that you normally would need to get from S3 (e.g. filesize, mimetype, author, timestamp, uuid). When you provide a URL to download the file, search locally and but provide a link to the S3 address.

Local file traversing is easy and this approach for S3 management is language agnostic. Local file traversing also avoids maintaining and querying a database of files or delays making a series of remote API calls to authenticate and get the bucket contents.

You could allow users to upload files directly to your server via FTP or HTTP and then transfer a batch of new and updated files to Amazon at off peak times by just recursing over the directories for files with any size. On the completion of a file transfer to Amazon, replace the web server file with an empty one of the same name. If a local file has any filesize then serve it directly because its awaiting batch transfer.

回答10:

Try this command:

aws s3api list-objects --bucket your-bucket --prefix sub-dir-path --output text --query 'Contents[].{Key: Key}'

Then you can pipe this into a grep to get specific file types to do whatever you want with them.

回答11:

If you're on Windows and have no time finding a nice grep alternative, a quick and dirty way would be:

aws s3 ls s3://your-bucket/folder/ --recursive > myfile.txt

and then do a quick-search in myfile.txt

The "folder" bit is optional.

P.S. if you don't have AWS CLI installed - here's a one liner using Chocolatey package manager

choco install awscli

P.P.S. If you don't have the Chocolatey package manager - get it! Your life on Windows will get 10x better. (I'm not affiliated with Chocolatey in any way, but hey, it's a must-have, really).

回答12:

Take a look at this documentation: http://docs.aws.amazon.com/AWSSDKforPHP/latest/index.html#m=amazons3/get_object_list

You can use a Perl-Compatible Regular Expression (PCRE) to filter the names.

回答13:

The way I did it is: I have thousands of files in s3. I saw the properties panel of one file in the list. You can see the URI of that file and I copy pasted that to the browser - it was a text file and it rendered nicely. Now I replaced the uuid in the url with the uuid that I had at hand and boom there the file is.

I wish AWS had a better way to search a file, but this worked for me.

回答14:

I did something as below to find patterns in my bucket

def getListOfPrefixesFromS3(dataPath: String, prefix: String, delimiter: String, batchSize: Integer): List[String] = {
    var s3Client = new AmazonS3Client()
    var listObjectsRequest = new ListObjectsRequest().withBucketName(dataPath).withMaxKeys(batchSize).withPrefix(prefix).withDelimiter(delimiter)
    var objectListing: ObjectListing = null
    var res: List[String] = List()

    do {
      objectListing = s3Client.listObjects(listObjectsRequest)
      res = res ++ objectListing.getCommonPrefixes
      listObjectsRequest.setMarker(objectListing.getNextMarker)
    } while (objectListing.isTruncated)
    res
  }

For larger buckets this consumes too much of time since all the object summaries are returned by the Aws and not only the ones that match the prefix and the delimiter. I am looking for ways to improve the performance and so far i've only found that i should name the keys and organise them in buckets properly.

回答15:

Status 2018-07: Amazon do have native sql like search for csv and json files!

https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/