I know how to expire objects in an S3 bucket using object expiration rules given a certain prefix, however for my purposes, I would like to set the expiry date programmatically on a per object basis.
The Java SDK seems to indicate that this is possible as it has a setExpirationTime method, however whenever I set an expiration Date using this method, nothing seems to happen and the object never expires. Additionally, looking at the object properties through the aws console, no expiry appears to be set.
Is per file expiration not supported ? / Are there any extra steps I need to do to get it to work ? / If per file expiration is not supported, is it possible to exclude a file that matches an expiration prefix from being expired ?
Thanks in advance!
It doesn't look like per-object expiration is supported, but rather a per-bucket lifecycle configuration with up to 100 rules per configuration, as you have found.
A bucket has one lifecycle configuration. A lifecycle configuration
can have up to 100 rules.
The lifetime value must be a nonzero positive integer. Amazon S3
calculates expiration time by adding the expiration period specified
in the rule to the object creation time and rounding the resulting
time to the next day midnight UTC.
If per file expiration is not supported, is it possible to exclude a
file that matches an expiration prefix from being expired ?
It doesn't look like you can overlap rules, either.
Take care to ensure the rules don't overlap. For example, the
following lifecycle configuration has a rule that sets objects with
the prefix "documents" to expire after 30 days. The configuration also
has another rule that sets objects with the prefix "documents/2011" to
expire after 365 days. In this case, Amazon S3 returns an error
message.
An alternative way to accomplish this task is to make use of object tags.
Set a unique tag for each object of the objects you want to set lifetimes for, Then create a life cycle configuration rule for each object and in the filter element you can use the object specific tag.
This tag can be the object name, Also you have the ability to make a rule for a subset of objects that don't have common prefix.
You can find more info about life cycle configuration here
http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html
Here's an alternative approach that might work for some situations:
You could set the bucket lifecycle to expire objects after a short number of days (1, 7, whatever). Then, before each object expires, programmatically extend its expiration if you don't want that object to expire yet.
You can reset the timestamp on an S3 object by "copying" the object to itself with an instruction to change the storage class to the same class it currently is. It's basically a "noop" that acts like a Linux/Unix "touch" command.
I wrote an article that walks through example aws-cli commands demonstrating how to do this: http://alestic.com/2013/09/s3-lifecycle-extend
I extended the expiration time by copying the object into itself using s3cmd command. It basically worked like 'touch'.
s3cmd cp s3://bucket-name/file.txt s3://bucket-name/file.txt
