S3: Generating Pre-Signed URL for a given Key. [Ke

Question

To get a message when the key is not present in S3 bucket. iam retrieving all the objects in that bucket and matching these keys with the given Search-key. if available returning the URL-String otherwise returning a message 'The specified key does not exist'.

Is their any other way to improve performance while accessing the key, which is not available in S3 bucket.

Here is my Code:

public class S3Objects {
    static Properties props = new Properties();
    static InputStream resourceAsStream;
    static {
        ClassLoader classLoader = new S3Objects().getClass().getClassLoader();
        resourceAsStream = classLoader.getResourceAsStream("aws.properties");
        try {
            props.load(resourceAsStream);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) throws IOException, AmazonServiceException, AmazonClientException, InterruptedException {
        AWSCredentials awsCreds = new 
                        BasicAWSCredentials(props.getProperty("accessKey"), props.getProperty("secretKey"));
                        // PropertiesCredentials(resourceAsStream);
        AmazonS3 s3Client = new AmazonS3Client( awsCreds );

        String s3_BucketName = props.getProperty("bucketname");
        String folderPath_fileName = props.getProperty("path");

        //uploadObject(s3Client, s3_BucketName, folderPath_fileName);
        //downloadObject(s3Client, s3_BucketName, folderPath_fileName);
        //getSignedURLforS3File(s3Client, s3_BucketName, folderPath_fileName);
        String url = getSingnedURLKey(s3Client, s3_BucketName, folderPath_fileName);
        System.out.println("Received response:"+url);
    }
    //  <MaxKeys>1000</MaxKeys>
    private static String getSingnedURLKey(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName) {
        String folderPath = folderPath_fileName.substring(0,folderPath_fileName.lastIndexOf("/"));      
        ObjectListing folderPath_Objects = s3Client.listObjects(s3_BucketName, folderPath);

        List<S3ObjectSummary> listObjects = folderPath_Objects.getObjectSummaries();
        for(S3ObjectSummary object : listObjects){
            if(object.getKey().equalsIgnoreCase(folderPath_fileName)){
                return getSignedURLforS3File(s3Client, s3_BucketName, folderPath_fileName);
            }
        }
        return "The specified key does not exist.";
    }

    //  providing pre-signed URL to access an object w/o any AWS security credentials.
   //   Pre-Signed URL = s3_BucketName.s3.amazonaws.com/folderPath_fileName?AWSAccessKeyId=XX&Expires=XX&Signature=XX
    public static String getSignedURLforS3File(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName){
        GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(s3_BucketName, folderPath_fileName, HttpMethod.GET);
        request.setExpiration( new Date(System.currentTimeMillis() + 1000 * 60 * 15) ); // Default 15 min

        String url = s3Client.generatePresignedUrl( request ).toString();
        System.out.println("Pre-Signed URL = " + url);
        return url;
    }

    public static void uploadObject(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName) 
            throws AmazonServiceException, AmazonClientException, InterruptedException{
        TransferManager tm = new TransferManager(s3Client);

        PutObjectRequest putObjectRequest = 
                new PutObjectRequest(s3_BucketName, folderPath_fileName, new File("newImg.jpg"));
        Upload myUpload = tm.upload( putObjectRequest );
        myUpload.waitForCompletion();//block the current thread and wait for your transfer to complete.
        tm.shutdownNow();            //to release the resources once the transfer is complete.
    }
   //   When accessing a key which is not available in S3, it throws an exception The specified key does not exist.
    public static void downloadObject(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName) throws IOException{
        GetObjectRequest request = new GetObjectRequest(s3_BucketName,folderPath_fileName);
        try{
            S3Object s3object = s3Client.getObject( request );
            System.out.println("Content-Type: " + s3object.getObjectMetadata().getContentType());
            S3ObjectInputStream objectContent = s3object.getObjectContent();

            FileUtils.copyInputStreamToFile(objectContent, new File("targetFile.jpg"));
        }catch(AmazonS3Exception s3){
            System.out.println("Received error response:"+s3.getMessage());
        }
    }

}

aws.properties

accessKey   =XXXXXXXXX
secretKey   =XXXXXXXXX

bucketname  =examplebucket
path        =/photos/2006/February/sample.jpg

Please let me know weather their is any other way to reduce no.of iterations over all the keys and get some message 'Key not exists'.

When am requesting a key to generate Pre-Signed URL. if

• Key Present « Returning the signed URL.
• Key Not Present « Message as key not available.

Answer 1

Amazon S3: Checking Key Exists and generating PresignedUrl

using getObjectMetadata():AmazonS3Client This is useful in obtaining only the object metadata, and avoids wasting bandwidth on fetching the object data.

private static String getPreSignedURLAsString_GetMetaData(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName) {
    GetObjectMetadataRequest request = new GetObjectMetadataRequest(s3_BucketName,folderPath_fileName);
    try{
        ObjectMetadata objectMetadata = s3Client.getObjectMetadata( request );
        System.out.println("Key Exists in S3");
        return getSignedURLforS3File(s3Client, s3_BucketName, folderPath_fileName);
    }catch(AmazonS3Exception s3){
        System.out.println("Received error response:"+s3.getMessage());
    }
    return "Key Not Found.";
}

using doesObjectExist():AmazonS3Client Internally it is making a call to getObjectMetadata(bucketName, objectName); - [aws-java-sdk-1.10.68]

private static String getPreSignedURLAsString_DoesKeyExists(AmazonS3 s3Client, String s3_BucketName, String folderPath_fileName) {
    try{
        boolean doesObjectExist = s3Client.doesObjectExist(s3_BucketName, folderPath_fileName);
        if (doesObjectExist) {
            System.out.println("Key Exists in S3");
            return getSignedURLforS3File(s3Client, s3_BucketName, folderPath_fileName);
        }
    }catch(AmazonS3Exception s3){
        System.out.println("Received error response:"+s3.getMessage());
    }
    return "Key Not Found.";
}

Required Jars[ aws-java{sdk, sdk-core, sdk-s3}, fasterxml.jackson{databind, core, annotations}, joda-time, http{clent, core}]

Answer 2

Use getObjectMetadata to quickly determine whether an object exists, given the key. If it succeeds, the object exists. If it doesn't, inspect the error to confirm it wasn't a transient error that needs to be retried. If not, there's no such key.

Iterating through the objects as you are doing not only doesn't scale, it'a also substantially more expensive, since the list requests carry a higher price per request than getting an object or getting its metadata, which should be very fast. This operation sends S3 an HTTP HEAD request, which returns 200 OK only if the object is there.

However, I would argue from a design perspective that this service shouldn't really care whether the object exists. Why would you receive requests for objects that don't exist? Who's asking for that? That should be the caller's problem -- and if you generate a signed URL for an object that doesn't exist, the request will fail with an error, when the caller tries to use the URL... But generating a signed URL for a non-existent object is a perfectly valid operation. The URL can be signed before the object is actually uploaded, and, as long as the URL hasn't expired, it will still work once the object is created, if it's created later.