How do I seed a mongo database using docker-compos

Question

I am trying to distribute a set of connected applications running in several linked containers that includes a mongo database that is required to:

• be distributed containing some seed data;
• allow users to add additional data.

Ideally the data will also be persisted in a linked data volume container.

I can get the data into the mongo container using a mongo base instance that doesn't mount any volumes (dockerhub image: psychemedia/mongo_nomount - this is essentially the base mongo Dockerfile without the VOLUME /data/db statement) and a Dockerfile config along the lines of:

ADD . /files
WORKDIR /files
RUN mkdir -p /data/db && mongod --fork --logpath=/tmp/mongodb.log && sleep 20 && \
mongoimport  --db testdb --collection testcoll  --type csv --headerline --file ./testdata.csv  #&& mongod --shutdown

where ./testdata.csv is in the same directory (./mongo-with-data) as the Dockerfile.

My docker-compose config file includes the following:

mongo:
  #image: mongo
  build: ./mongo-with-data
  ports:
    - "27017:27017"
  #Ideally we should be able to mount this against a host directory
  #volumes:
  #  - ./db/mongo/:/data/db
  #volumes_from:
  #  - devmongodata

#devmongodata:
#    command: echo created
#    image: busybox
#    volumes: 
#       - /data/db

Whenever I try to mount a VOLUME it seems as if the original seeded data - which is stored in /data/db - is deleted. I guess that when a volume is mounted to /data/db it replaces whatever is there currently.

That said, the docker userguide suggests that: Volumes are initialized when a container is created. If the container’s base image contains data at the specified mount point, that existing data is copied into the new volume upon volume initialization? So I expected the data to persist if I placed the VOLUME command after the seeding RUN command?

So what am I doing wrong?

The long view is that I want to automate the build of several linked containers, and then distribute a Vagrantfile/docker-compose YAML file that will fire up a set of linked apps, that includes a pre-seeded mongo database with a (partially pre-populated) persistent data container.

Answer 1

Here is a writeup of how we're using disposable containers to clean and seed images https://ardoq.com/delightful-database-seeding-with-docker/

Answer 2

To answer my own question:

• simple YAML file to create simple mongo container linked to a data volume container, fired up by Vagrant docker compose.
• in the Vagrantfile, code along the lines of:

config.vm.provision :shell, :inline => <<-SH docker exec -it -d vagrant_mongo_1 mongoimport --db a5 --collection roads --type csv --headerline --file /files/AADF-data-minor-roads.csv SH

to import the data.

Package the box.

Distribute the box.

For the user, a simple Vagrantfile to load the box and run a simple docker-compose YAML script to start the containers and mount the mongo db against the data volume container.

Answer 3

You can use this image that provides docker container for many jobs ( import, export , dump )

Look at the example using docker-compose

Answer 4

I do this using another docker container whose only purpose is to seed mongo, then exit. I suspect this is the same idea as ebaxt's, but when I was looking for an answer to this, I just wanted to see a quick-and-dirty, yet straightforward, example. So here is mine:

docker-compose.yml

mongodb:
  image: mongo
  ports:
    - "27017:27017"

mongo-seed:
  build: ./mongo-seed
  links:
    - mongodb

# my webserver which uses mongo (not shown in example)
webserver:
  build: ./webserver
  ports:
    - "80:80"
  links:
    - mongodb

mongo-seed/Dockerfile

FROM mongo

COPY init.json /init.json
CMD mongoimport --host mongodb --db reach-engine --collection MyDummyCollection --type json --file /init.json --jsonArray

mongo-seed/init.json

[
  {
    "name": "Joe Smith",
    "email": "[e-mail]",
    "age": 40,
    "admin": false
  },
  {
    "name": "Jen Ford",
    "email": "[e-mail]",
    "age": 45,
    "admin": true
  }
]

Answer 5

You can use Mongo Seeding Docker image.

Why?

• You have the Docker image ready to go
• You are not tied to JSON files - JavaScript and TypeScript files are supported as well (including optional model validation with TypeScript)

Example usage with Docker Compose:

version: '3'
services:
  database:
    image: 'mongo:3.4.10'
    ports:
    - '27017:27017'
  api:
    build: ./api/
    command: npm run dev
    volumes: 
    - ./api/src/:/app/src/
    ports:
    - '3000:3000'
    - '9229:9229'
    links:
    - database
    depends_on:
    - database
    - data_import
    environment: 
    - &dbName DB_NAME=dbname
    - &dbPort DB_PORT=27017 
    - &dbHost DB_HOST=database
  data_import:
    image: 'pkosiec/mongo-seeding:3.0.0'
    environment:
    - DROP_DATABASE=true
    - REPLACE_ID=true
    - *dbName
    - *dbPort
    - *dbHost
    volumes:
    - ./data-import/dev/:/data-import/dev/
    working_dir: /data-import/dev/data/
    links:
    - database
    depends_on:
    - database

Disclaimer: I am the author of this library.

Answer 6

I have found useful to use Docker Custom Images and using volumes, instead of creating another container for seeding.

File Structure

.
├── docker-compose.yml
├── mongo
│   ├── data
│   ├── Dockerfile
│   └── init-db.d
│       └── seed.js

Every File location mentioned in Dockerfile/docker-compose.yml, is relative to location of docker-compose.yml

DOCKERFILE

FROM mongo:3.6

COPY ./init-db.d/seed.js /docker-entrypoint-initdb.d

docker-compose.yml

version: '3'

services:
  db:
    build: ./mongo
    restart: always
    volumes:
      - ./mongo/data:/data/db #Helps to store MongoDB data in `./mongo/data`
    environment:
      MONGO_INITDB_ROOT_USERNAME: {{USERNAME}}
      MONGO_INITDB_ROOT_PASSWORD: {{PWD}}
      MONGO_INITDB_DATABASE: {{DBNAME}}

seed.js

// Since Seeding in Mongo is done in alphabetical order... It's is important to keep
// file names alphabetically ordered, if multiple files are to be run.

db.test.drop();
db.test.insertMany([
  {
    _id: 1,
    name: 'Tensor',
    age: 6
  },
  {
    _id: 2,
    name: 'Flow',
    age: 10
  }
])

docker-entrypoint-initdb.d can be used for creating different users and mongodb administration related stuffs, just create an alphabetical ordered named js-script to createUser etc...

For more details on how to customize MongoDB Docker service, read this

Also, it is good to keep your passwords and usernames secure from Public, DO NOT push credentials on public git, instead use Docker Secrets. Also read this Tutorial on Secrets

Do note, it is not necessary to go into docker-swarm mode to use secrets. Compose Files supports secrets as well. Check this

Secrets can also be used in MongoDB Docker Services