I've been surprised to find out that file deletion was not replicated in a S3 bucket Cross Region Replication situation, running this simple test:
- simplest configuration of a CRR
- upload a new file
- check it is replicated
- delete the file (not a version of the file)
So I checked the documentation and I find this statement:
If you delete an object from the source bucket, the following occurs:
- If you make a DELETE request without specifying an object version ID, Amazon S3 adds a delete marker. Amazon S3 deals with the delete
marker as follows:
- If using latest version of the replication configuration, that is you specify the Filter element in a replication configuration rule,
Amazon S3 does not replicate the delete marker.
- If don't specify the Filter element, Amazon S3 assumes replication configuration is a prior version V1. In the earlier
version, Amazon S3 handled replication of delete markers differently.
For more information, see Backward Compatibility .
The later link to backward compat tell me that:
- When you delete an object from your source bucket without specifying an object version ID, Amazon S3 adds a delete marker. If you use V1 of the replication configuration XML, Amazon S3 replicates delete markers that resulted from user actions.[...]
In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled.
So if I sum this up:
- CRR configuration is considered v1 if there is no Filter
- with CRR configuration v1, file deletion is replicated, not with v2
Well, this is my configuration :
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::271226720751:role/service-role/s3crr_role_for_mybucket_to_myreplica",
"Rules": [
{
"ID": "first replication rule",
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::myreplica"
}
}
]
}
}
And deletion is not replicated. So it makes me think that my configuration is still considered V2 (even if I have no filter).
So, can someone confirm this presumption?
And could someone tell me what does:
In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled
really mean?
I have seen exactly the same behaviour. I was unable to create a v1 situation to get DeleteMarker replication to occur.
The issue comes from still not clear documentation from AWS.
To use DeleteMarkerReplication, you need V1 of the configuration. To let AWS know that you want V1, you need to specify a Prefix element in your configuration, and no DeleteMarkerReplication element, so your first try was almost correct.
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::271226720751:role/service-role/s3crr_role_for_mybucket_to_myreplica",
"Rules": [
{
"ID": "first replication rule",
"Prefix": "",
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::myreplica"
}
}
]
}
}
And of course you need the s3:ReplicateDelete permission in your policy.
I believe I've figured this out. It looks like whether the Delete Markers are replicated or not depends on the permissions in the Replication Role.
If your replication role has the permission s3:ReplicateDelete on the destination, then Delete Markers will be replicated. If if does not have that permission they are not.
Below is the Cloudformation YAML for my Replication role with the ReplicateDelete permission commented out as an example. With this setup it does not replicate Delete Markers, uncomment the permission and it will. Note the permissions is based on what AWS actually creates if you set up the replication via the console (and they differ slightly from those in the documentation).
ReplicaRole:
Type: AWS::IAM::Role
Properties:
#Path: "/service-role/"
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- s3.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: "replication-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Resource:
- !Sub "arn:aws:s3:::${LiveBucketName}"
- !Sub "arn:aws:s3:::${LiveBucketName}/*"
Action:
- s3:Get*
- s3:ListBucket
- Effect: Allow
Resource: !Sub "arn:aws:s3:::${LiveBucketName}-replica/*"
Action:
- s3:ReplicateObject
- s3:ReplicateTags
- s3:GetObjectVersionTagging
#- s3:ReplicateDelete
Adding a comment as an answer because I cannot comment to @john-eikenberry's answer. I have tested answer suggested by John (Action "s3:ReplicateDelete") but it is not working.
Edit: A failed attempt:
I have also tried to put bucket replication with delete marker enabled but it failed. Error message is:
An error occurred (MalformedXML) when calling the PutBucketReplication
operation: The XML you provided was not well-formed or did not
validate against our published schema
Experiment details:
Existing replication configuration:
aws s3api get-bucket-replication --bucket my-source-bucket > my-source-bucket.json
{
"Role": "arn:aws:iam::account-number:role/s3-cross-region-replication-role",
"Rules": [
{
"ID": " s3-cross-region-replication-role",
"Priority": 1,
"Filter": {},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::my-destination-bucket"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
aws s3api put-bucket-replication --bucket my-source-bucket --replication-configuration file://my-source-bucket-updated.json
{
"Role": "arn:aws:iam::account-number:role/s3-cross-region-replication-role",
"Rules": [
{
"ID": " s3-cross-region-replication-role",
"Priority": 1,
"Filter": {},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::my-destination-bucket"
},
"DeleteMarkerReplication": {
"Status": "Enabled"
}
}
]
}
