Why does the docker-compose healthcheck of my mong

Question

I'm using docker-compose to stand up an Express/React/Mongo app. I can currently stand up everything using retry logic in the express app. However, I would prefer to use Docker's healthcheck to prevent the string of errors when the containers initially spin up. However, when I add a healthcheck in my docker-compose.yml, it hangs for the interval/retry time limit and exits with:

ERROR: for collector  Container "70e7aae49c64" is unhealthy.

ERROR: for server  Container "70e7aae49c64" is unhealthy.
ERROR: Encountered errors while bringing up the project.

It seems that my healthcheck never returns a healthy status, and I'm not entirely sure why. The entirety of my docker-compose.yml:

version: "2.1"
services:
  mongo:
    image: mongo
    volumes:
      - ./data/mongodb/db:/data/db
    ports:
      - "${DB_PORT}:${DB_PORT}"
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongo mongo:27017/test --quiet 1
      interval: 10s
      timeout: 10s
      retries: 5
  collector:
    build: ./collector/
    environment:
      - DB_HOST=${DB_HOST}
      - DB_PORT=${DB_PORT}
      - DB_NAME=${DB_NAME}
    volumes:
      - ./collector/:/app
    depends_on:
      mongo:
        condition: service_healthy
  server:
    build: .
    environment:
      - SERVER_PORT=$SERVER_PORT
    volumes:
      - ./server/:/app
    ports:
      - "${SERVER_PORT}:${SERVER_PORT}"
    depends_on:
      mongo:
        condition: service_healthy

For the test, I've also tried:

["CMD", "nc", "-z", "localhost", "27017"] 

And:

["CMD", "bash", "/mongo-healthcheck"]

I've also tried ditching the healthcheck altogether, following the advice of this guy. Everything stands up, but I get the dreaded errors in the output before a successful connection:

collector_1  | MongoDB connection error: MongoNetworkError: failed to connect to server [mongo:27017] on first connect [MongoNetworkError: connect 
ECONNREFUSED 172.21.0.2:27017]
collector_1  | MongoDB connection with retry
collector_1  | MongoDB connection error: MongoNetworkError: failed to connect to server [mongo:27017] on first connect

The ultimate goal is a clean startup output when running the docker-compose up --build. I've also looked into some of the solutions in this question, but I haven't had much luck with wait-for-it either. What's the correct way to wait for Mongo to be up and running before starting the other containers, and achieving a clean startup?

Answer 1

Firstly, I'd suggest to update the docker-compose.yaml file version to at least 3.4 (version: "3.5"), then please add the start_period option to your mongo healthcheck

Note: start_period is only supported for v3.4 and higher of the compose file format.

start period provides initialization time for containers that need time to bootstrap. Probe failure during that period will not be counted towards the maximum number of retries. However, if a health check succeeds during the start period, the container is considered started and all consecutive failures will be counted towards the maximum number of retries.

So it would look something like this:

healthcheck:
  test: echo 'db.runCommand("ping").ok' | mongo mongo:27017/test --quiet
  interval: 10s
  timeout: 10s
  retries: 5
  start_period: 40s

Answer 2

We can use MongoDB's serverStatus command to do the health check, as the MongoDB document puts it this way:

Monitoring applications can run this command at a regular interval to collect statistics about the instance.

Because this command serverStatus requires authentication, you need setup the health check similar to the configuration shown below:

version: '3.4'

services:
  mongo:
    image: mongo
    restart: always
    healthcheck:
      test: echo 'db.runCommand({serverStatus:1}).ok' | mongo admin -u $MONGO_INITDB_ROOT_USERNAME -p $MONGO_INITDB_ROOT_PASSWORD --quiet | grep 1
      interval: 10s
      timeout: 10s
      retries: 3
      start_period: 20s
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example

That's it. If your MongoDB instance is healthy, you will see something similar to mine:

$ docker ps
CONTAINER ID  IMAGE  COMMAND                 CREATED         STATUS                   PORTS      NAMES
01ed0e02aa70  mongo  "docker-entrypoint.s…"  11 minutes ago  Up 11 minutes (healthy)  27017/tcp  demo_mongo_1

Answer 3

When i execute the echo db.runCommand("ping").ok' | mongo localhost:27017/test --quiet 1 command in the docker container, the result is:

    2019-04-19T02:39:19.770+0000 E -        [main] file [1] doesn't exist
    failed to load: 1

Try this

healthcheck:
  test: bash -c "if mongo --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)'; then exit 0; fi; exit 1;"

Answer 4

I found a solution here https://github.com/docker-library/healthcheck/tree/master/mongo

Note, it explains why health check is not included into official image https://github.com/docker-library/cassandra/pull/76#issuecomment-246054271

docker-healthcheck

#!/bin/bash
set -eo pipefail

if mongo --quiet "localhost/test" --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)'; then
    exit 0
fi

exit 1

In the example from the link, they use host variable

host="$(hostname --ip-address || echo '127.0.0.1')"

if mongo --quiet "$host/test" --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)'; then
# continues the same code

It did not work for me, so I replaced the host with localhost.

In docker-compose

mongo:
  build:
    context: "./mongodb"
    dockerfile: Dockerfile
  container_name: crm-mongo
  restart: always
  healthcheck:
    test:  ["CMD", "docker-healthcheck"]
    interval: 10s
    timeout: 2s
    retries: 10

Alternatively, you can execute health checks in container. Change Dockerfile or that.

FROM mongo:4

ADD docker-healthcheck /usr/local/bin/