How do I update metadata for an existing Amazon S3

Question

I need to update the cache control header in all AmazonS3's Cloud Files. However, I can't figure out how I do that using the jclouds API. I'm using apache jclouds plugin. And I got two related answers:

• jclouds : how do I update metadata for an existing blob?
• Set Expires header for an existing S3 object using AWS Java SDK

The first answer is suggesting to use SwiftKey Api class which is not available in grails's jcloud plugin. The second answer is using AWS java sdk for which there is already a grails wrapping plugin https://grails.org/plugin/aws-sdk but it doesn't support metadata update.

Answer 1

As at March 2018, the latest version of the AWS SDK for .NET Core has changed. It now uses asynchronous programming. Many of the method signatures have changed. While you still cannot change metadata without the object copy solution that Dan has suggested, the code to do so has.

My solution is to update the existing S3 object with the modified metadata.

The following works for me to update a single metadata value (based on key and new value). I've got two loops for setting the metadata but it could be optimised to just have one:

string fileContents = string.Empty;
Dictionary<string, string> fileMetaData = null;

GetObjectRequest request = new GetObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName
};

var response = await s3Client.GetObjectAsync(request);

// Read the contents of the file
using (var stream = response.ResponseStream)
{
  // Get file contents
  TextReader tr = new StreamReader(stream);
  fileContents = tr.ReadToEnd();
}

// Create the File Metadata collection
fileMetaData = new Dictionary<string, string>();
foreach (string metadataKey in response.Metadata.Keys)
{
  fileMetaData.Add(metadataKey, response.Metadata[metadataKey]);
}

// Update the metadata value (key to update, new value)
fileMetaData[metaDataKeyToUpdate] = metaDataNewValue;

// Update the existing S3 object    
PutObjectRequest putRequest1 = new PutObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName,
  ContentBody = fileContents
};

// Set the metadata
foreach (string metadataKey in response.Metadata.Keys)
{
  putRequest1.Metadata.Add(metadataKey, fileMetaData[metadataKey]);
}

PutObjectResponse response1 = await s3Client.PutObjectAsync(putRequest1);

Answer 2

It is possible to change the metadata by performing an object copy (see How to update metadata using Amazon S3 SDK):

ObjectMetadata metadataCopy = new ObjectMetadata();
// copy previous metadata
metadataCopy.addUserMetadata("newmetadata", "newmetadatavalue");

CopyObjectRequest request = new CopyObjectRequest(bucketName, existingKey, bucketName, existingKey)
      .withNewObjectMetadata(metadataCopy);

amazonS3Client.copyObject(request);

Whether this is philosophically an "update" is up to you to decide.

Answer 3

You can't:

Each Amazon S3 object has data, a key, and metadata. Object key (or key name) uniquely identifies the object in a bucket. Object metadata is a set of name-value pairs. You can set object metadata at the time you upload it. After you upload the object, you cannot modify object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata.

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Answer 4

In my case, I care about the object versions, and copying creates a new version. I decided to create a parallel, "hidden" object with the mutable metadata. For example, if I have /foo/bar/baz, then I would create /foo/bar/.baz.

Answer 5

PHP Example

I understand this question was not PHP specific, but it may help someone as it is a top result on Google.

This will overwrite the existing object.

$client = new \Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'BUCKET-REGION'
]);

$updateResponse = $client->copyObject([
    'Key' => 'OBJECTKEY',
    'Bucket' =>  'MYBUCKET',
    'CopySource' => 'MYBUCKET/OBJECTKEY',
    'MetadataDirective' => 'REPLACE',
    'Metadata' => [
        'width' => 441,
        'height' => 189
    ]
]);