Retrieve static file from amazom s3 bucket

Question

I am trying to configure my nginx in such a way that whenever there is some bad gateway response, I try to fetch static html contents from the s3 bucket.

The url structure of the request is some_bucket/folder1/folder2/text

And the data is stored in s3 bucket with directory structure as s3.amazonaws.com/some_bucket/folder1/folder2/folder1_folder2.html

I am not able to determine the values for folder1 and folder2 so that I can make the html file dynamically and use proxy_pass. Also, I tried try_files but I think that does not work for urls.

Any idea how to tackle this problem.

Thanks.

Answer 1

Nginx S3 proxy can handle dynamically built URL, you can also hide a directory and even part of private URL such AWS Key:

For instance the basis URL is the following:

https://your_bucket.s3.amazonaws.com/readme.txt?AWSAccessKeyId=YOUR_ONLY_ACCESS_KEY&Signature=sagw4gsafdhsd&Expires=3453445231

Resulted URL:

https://your_server/proxy_private_file/readme.txt?st=sagw4gsafdhsd&e=3453445231

The configuration is not difficult:

location ~* ^/proxy_private_file/(.*) {
  set $s3_bucket        'your_bucket.s3.amazonaws.com';
  set $aws_access_key   'AWSAccessKeyId=YOUR_ONLY_ACCESS_KEY';
  set $url_expires      'Expires=$arg_e';
  set $url_signature    'Signature=$arg_st';
  set $url_full         '$1?$aws_access_key&$url_expires&$url_signature';

  proxy_http_version     1.1;
  proxy_set_header       Host $s3_bucket;
  proxy_set_header       Authorization '';
  proxy_hide_header      x-amz-id-2;
  proxy_hide_header      x-amz-request-id;
  proxy_hide_header      Set-Cookie;
  proxy_ignore_headers   "Set-Cookie";
  proxy_buffering        off;
  proxy_intercept_errors on;

  proxy_pass             http://$s3_bucket/$url_full;  
}

See the full configuration for more details.

Answer 2

This is what I did for someone(probably newbie) who may encounter this problem.

 location ~* ^/some_bucket/(.*)/(.*)/.* {
      proxy_pass http://s3.amazonaws.com/some_bucket/$1/$2/$1_$2.html;
 }

~* means case insensitive regex match ^ means anything before () for catching parameters.

For example, User enters www.example.com/some_bucket/folder1/folder2/text

Then, it is processed as,

~* ensures case insensitive search(for case sensitive skip *(means just put ~))

^ matches www.example.com.

/some_bucket/ is matched then,

.* means any number of any character(for any numeric, replace with [0-9]*)

() ensures that matched values gets catched

So, $1 catches folder1

$2 catches folder2

Then

.* without parenthesis matches any charater but does not catch the matched value

Now the catched values can be used to find the file in amazon bucket using

proxy_pass http://s3.amazonaws.com/some_bucket/$1/$2/$1_$2.html

https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms can be helpful