In Brief

In order to keep the uploaded media (S3 objects) private for all the clients on my multi-tenant system I implemented a Cloudfront CDN deployment and configured it (and its Origin S3 Bucket) to force the use of signed URLs in order to GET any of the objects.

The Method

First, the user is authenticated via my system, and then a signed URL is generated and returned to them using the AWS.CloudFront.Signer.getSignedUrl() method provided by the AWS JS SDK. so they can make the call to CF/S3 to download the object (image, PDF, docx, etc). Pretty standard stuff.

The Problem

The above method works 95% of the time. The user obtains a signed URL from my system and then when they make an XHR to GET the object it's retrieved just fine.

But, 5% of the time a 403 is thrown with a CORS error stating that the client origin is not allowed by Access-Control-Allow-Origin.

This bug (error) has been confirmed across all environments: localhost, dev.myapp.com, prod.myapp.com. And across all platforms/browsers.

There's such a lack of rhyme or reason to it that I'm actually starting to think this is an AWS bug (they do happen, from time-to-time).

The Debugging Checklist So Far

I've been going out of my mind for days now trying to figure this out. Here's what I've attempted so far:

Have you tried a different browser/platform?

Yes. The issue is present across all client origins, browsers (and versions), and all platforms.

Is your S3 Bucket configured for CORS correctly?

Yes. It's wide-open in fact. I've even set <MaxAgeSeconds>0</MaxAgeSeconds> in order to prevent cacheing of any pre-flight OPTIONS requests by the client:

Is the signed URL expired?

Nope. All of the signed URLs are set to expire 24hrs after generation. This problem has shown up even seconds after any given signed URL is generated.

Is there an issue with the method used to generate the signed URLs?

Unlikely. I'm simply using the AWS.CloudFront.Signer.getSignedUrl() method of their JS SDK. The signed URLs do work most of the time, so it would seem very strange that it would be an issue with the signing process. Also, the error is clearly a CORS error, not a signature mis-match error.

Is it a timezone/server clock issue?

Nope. The system does serve users across many timezones, but that theory proved to be false given that the signed URLs are all generated on the server-side. The timezone of the client doesn't matter, it gets a signed URL good for 24hrs from the time of generation no matter what TZ it's in.

Is your CF distro configured properly?

Yes, so far as I can make out by following several AWS guides, tutorials, docs and such.

Here's a screenshot for brevity. You can see that I've disabled cacheing entirely in an attempt to rule that out as a cause:

Are you seeing this error for all mime-types?

No. This error hasn't been seen for any images, audio, or video files (objects). With much testing already done, this error only seems to show up when attempting to GET a document or PDF file (.doc, .docx, .pdf). This lead me to believe that this was simply an Accept header mis-match error: The client was sending an XHR with the the header Accept: pdf, but really the signature was generated for Accept: application/pdf. I haven't yet been able to fully rule this out as a cause. But it's highly unlikely given that the errors are intermittent. So if it were a Accept header mis-match problem then it should be an error every time.

Also, the XHR is sending Accept: */* so it's highly unlikely this is where the issue is.

The Question

I've really hit a wall on this one. Can anyone see what I'm missing here? The best I can come up with is that this is some sort of "timing" issue. What sort of timing issue, or if it even is a timing issue, I've yet to figure out.

Thanks in advance for any help.

Found the solution for the same on serverfault.

https://serverfault.com/questions/856904/chrome-s3-cloudfront-no-access-control-allow-origin-header-on-initial-xhr-req

You apparently cannot successfully fetch an object from HTML and then successfully fetch it again with as a CORS request with Chrome and S3 (with or without CloudFront), due to peculiarities in the implementations.

Adding the answer from original post so that it does not get lost.

Workaround:

This behavior can be worked-around with CloudFront and Lambda@Edge, using the following code as an Origin Response trigger.

This adds Vary: Access-Control-Request-Headers, Access-Control-Request-Method, Origin to any response from S3 that has no Vary header. Otherwise, the Vary header in the response is not modified.

'use strict';

// If the response lacks a Vary: header, fix it in a CloudFront Origin Response trigger.

exports.handler = (event, context, callback) => {
    const response = event.Records[0].cf.response;
    const headers = response.headers;

    if (!headers['vary'])
    {
        headers['vary'] = [
            { key: 'Vary', value: 'Access-Control-Request-Headers' },
            { key: 'Vary', value: 'Access-Control-Request-Method' },
            { key: 'Vary', value: 'Origin' },
        ];
    }
    callback(null, response);
};