Retrieve S3 file as Object instead of downloading

问题:

I just started learning and using S3, read the docs. Actually I didn't find anything to fetch the file into an object instead of downloading it from S3? if this could be possible, or I am missing something?

Actually I want to avoid additional IO after downloading the file.

回答1:

You could use StringIO and get file content from S3 using get_contents_as_string, like this:

import pandas as pd
import StringIO
from boto.s3.connection import S3Connection

AWS_KEY = 'XXXXXXDDDDDD'
AWS_SECRET = 'pweqory83743rywiuedq'
aws_connection = S3Connection(AWS_KEY, AWS_SECRET)
bucket = aws_connection.get_bucket('YOUR_BUCKET')

fileName = "test.csv"

content = bucket.get_key(fileName).get_contents_as_string()
reader = pd.read_csv(StringIO.StringIO(content))

回答2:

You might be looking for the get_object() method of the boto3 S3 client:

http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.get_object

This will get you a response object dictionary with member Body that is a StreamingBody object, which you can use as normal file and call .read() method on it. To get the entire content of the S3 object into memory you would do something like this:

s3_client = boto3.client('s3')
s3_response_object = s3_client.get_object(Bucket=BUCKET_NAME_STRING, Key=FILE_NAME_STRING)
object_content = s3_response_object['Body'].read()

回答3:

I prefer this approach, equivalent to a previous answer:

import boto3
s3 = boto3.resource('s3')
def read_s3_contents(bucket_name, key):
    response = s3.Object(bucket_name, key).get()
    return response['Body'].read()

But another approach could read the object into StringIO:

import StringIO
import boto3
s3 = boto3.resource('s3')
def read_s3_contents_with_download(bucket_name, key):
    string_io = StringIO.StringIO()
    s3.Object(bucket_name, key).download_fileobj(string_io)
    return string_io.getvalue()