How to mock Amazon S3 in an integration test

Question

I'm trying to get an "walking skeleton" of my app up that will use S3 for persistence. I would like to use a fake S3 service so each developer's desktop can read/write at will.

I thought mocks3 would be perfect, as I could get a jetty server up in my jUnit tests. The problem is that mocks3 doesn't allow any writes. Not even to set it up as far as I can tell.

So how do others do this?

Answer 1

You can use scality s3server, in can run on your machine either using node.js or via docker and it gives you a local S3 service instance. It's open source under a BSD license github.com/scality/s3

Answer 2

Have a look at Adobe's S3Mock. This S3 Mock server can be started via a Docker container or JUnit 4/5 rules.

Answer 3

If you're ok with depending on a running docker container, and want something well-supported, you could consider using localstack

Before running your tests, start S3 like so:

docker run --name localstack -d -p 5000:5000 -e SERVICES=s3:5000 localstack/localstack

And then stop it when tests complete like so:

docker stop localstack

You'll need to configure your s3 client to point to localhost:5000 for tests. In java, this can be done like so:

        AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
                 "http://localhost:5000", 
                 "us-west-2"))
            .build();

Answer 4

Tornado, a python web framework, has an example app that is just what you're looking for.

https://github.com/facebook/tornado/blob/master/demos/s3server/s3server.py

It can be used out of the box.

Answer 5

One option is to scrap the jetty server and use Apache VFS and the S3 plugin. With that, you can use the memory or file-based storage implementations for the integration testing.

Answer 6

Another option is S3 ninja - emulates the S3 API for development and testing purposes.