Docker 1.12: Multiple replicas, single database

Question

With the introduction of the new 'swarm mode' with Docker 1.12, we've been trying to migrate our application on containers and make use of the swarm mode's orchestration & clusters.

Our application requires some initial database scripts to be run for it to start. We're not packaging the database inside our dockerized application so that it could follow a stateless microservice architecture and multiple containers would eventually talk to a single (at the moment) database instance.

While creating the service, we cannot use --replicas with the create service command as multiple instances would try and create tables on a single database and fail. Although our scripts would check if the database has been set-up and skip the creation but since all containers start simultaneously, it could not be used.

We couldn't find any wait-for kind of mechanism that we could leverage with dockers for this issue. It would have been good if we could only start the second container when the first one had created the database (and exposed the ports) but how can we configure inter-container communication for this?

Alternatively, can tools like flywaydb help in some way?

How should this be used in production?

Answer 1

From the Flyway FAQ:

Can multiple nodes migrate in parallel? Yes! Flyway uses the locking technology of your database to coordinate multiple nodes. This ensures that even if even multiple instances of your application attempt to migrate the database at the same time, it still works. Cluster configurations are fully supported.

Answer 2

There is no easy way to coordinate this among containers. It basically requires a distributed lock solution. The first container that gets the lock could create db, while, other containers that not get the lock need to wait.

In AWS, you could leverage DynamoDB for it. DynamoDB supports conditional update. The container first tries to create the lock key in DynamoDB with "attribute_not_exists(yourKey)". The first creation will succeed and other creations will be rejected. The first container needs to create another key in DynamoDB to indicate the db is ready. Other containers simply waits till the ready key is created.

Or you could do it in your service deployment script. The script could create the service with 1 replica. Then keep checking if db is created. If yes, scale the service, such as docker service update yourservicce --replicas 5.