AWS error downloading object from S3, “profile fil

Question

I've already seen this, but there was no answer to explain my problem. I first used the sample provided here (GetObject class), and it worked immediately on my desktop. However, my friend could not get it to work on his machine, nor will it work on our EC2 instance.

It was mentioned that there are to be credentials files specified, which makes sense, but I never had to do that and am pretty sure the default permissions were set to enable accessing this bucket.

Here's the stacktrace:

Exception in thread "main" java.lang.IllegalArgumentException: profile file cannot be null
    at com.amazonaws.util.ValidationUtils.assertNotNull(ValidationUtils.java:37)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:142)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:133)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:100)
    at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:135)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1029)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1049)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035)
    at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4474)
    at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4448)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4020)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1307)
    at GetObject.main(GetObject.java:26)

I can guarantee that neither the bucketName nor the key params in the GetObjectRequest are null. What's the discrepancy here? Why might it succeed only on my PC? Is this at all related to the fact that I had to supplement numerous jars that the aws-sdk jar was supposed to have already (jackson-databind, jackson-core, jackson-annotations, httpclient, httpcore, commons-logging, and joda-time)? It seems similar, what with the otherwise inexplicable errors (giving non-null params, something in aws-sdk says it's null).

Answer 1

It looks like you solved this in the comments, but I got burned on this and want to leave a clearer answer for future readers. To be super clear, the problem here has nothing to do with files in S3. This error message has nothing to do with the file on your hard drive nor the file that you're trying to push/pull from S3. The problem is that you're initializing S3 with something like:

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());

When you do that, it looks in ~/.aws/credentials for a list of profiles. This might work great on your computer but won't work anywhere that you're getting AWS access through an IAM role (ex. Lambda, Docker, EC2 instance, etc). The fix, is to initialize the AmazonS3Client like:

AmazonS3 s3Client = new AmazonS3Client();

If you're using code that requires some kind of credentials provider, you can also do:

AmazonS3 s3Client = new AmazonS3Client(DefaultAWSCredentialsProviderChain.getInstance());

Hopefully that helps the next person. In my case, I was using DynamoDB and SQS, but I had the same error. I originally ignored this question because I thought your problem was S3 related and was super confused. Wrist slapped.

Answer 2

The reason is that the computer of your friend doesn't have the "credentials" file.

To solve the problem, create file:

C:\Users\USERNAME \.aws\credentials"

for Windows, or create file:

~/.aws/credentials

for macOS, Linux or Unix. And then write

aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key"

into the file.

Or, you can also set the environment variable of aws_access_key_id and aws_secret_access_key.

Answer 3

The top voted answer by Ryan put me on the right track, but since AmazonS3Client is now deprecated, this code has resolved the problem for me

    AmazonS3 s3 = AmazonS3ClientBuilder.standard()
                  .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
                  .build();

This code appears to correctly pick up the active IAM role, say in Lambda.