EC2 instance role gets 'Unknown' error whe

2019-05-27 10:39发布

问题:

I've got an ASG that assigns an IAM Role to each of the instances that join it. Therefore, each instance has the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables baked-in, which will be used upon instantiation to download and decrypt credentials that are stored in an S3 bucket and encrypted using KMS keys.

So I'll have the following components:

  • An S3 bucket called top-secret.myapp.com
  • All objects in this bucket are encrypted using a KMS key called My-KMS-Key
  • An IAM instance role with inline policies attached granting it the ability to interact with both the bucket and the KMS key used to encrypt/decrypt the contents of the bucket (see below)
  • A user data script that installs the aws-cli upon instantiation and then goes about attempting to download and decrypt an object from the top-secret.myapp.com bucket.

The User Data Script

Upon instantiation, any given instance runs the following script:

#!/bin/bash

apt-get update
apt-get -y install python-pip
apt-get -y install awscli

cd /home/ubuntu
aws s3 cp s3://top-secret.myapp.com/secrets.sh . --region us-east-1
chmod +x secrets.sh
. secrets.sh
shred -u -z -n 27 secrets.sh


IAM Role Policies

The IAM role for my ASG instances has three policies attached inline:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::top-secret.myapp.com"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::top-secret.myapp.com/secrets.sh"
            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": "arn:aws:kms:us-east-1:UUID-OF-MY-SECRET-KEY-HERE"
        }
    ]
}

The first policy is essentially a full-root-access policy with no restrictions. Or so I thought, but it doesn't work. So I thought it might be that I need to explicitly apply policies that allow interaction with S3 encryption and/or KMS, makes sense.

So I added the second policy that allows the IAM instance role to list the top-secret.myapp.com bucket, and LIST and GET the secrets.sh object within the bucket. But this produced the error illustrated below.

The (Unknown) Error I'm Getting

download failed: s3://top-secret.myapp.com/secrets.sh to ./secrets.sh
A client error (Unknown) occurred when calling the GetObject operation: Unknown

Anyone have any idea what could be causing this error?

Note: This method for transferring encrypted secrets from S3 and decrypting them on-instance works fine using the standard Amazon S3 service master key

回答1:

For me, the issue was two-fold:

  1. If you're using server-side encryption via KMS, you need to supply the --sse aws:kms flag to the aws s3 cp [...] command.
  2. I was installing an out-of-date version of awscli (version 1.2.9) via apt, and that version didn't recognize the --sse aws:kms command
    • Running apt-get remove awscli and installing via pip install awscli gave me version 1.10.51, which worked.

EDIT:

If you're using a different KMS key than the default master key for your account, you will need to also add the following flag:

--sse-kms-key-id [YOUR KMS KEY ID]