How to automatically update your docker containers

Question

Say I have a trivial container based on the ubuntu:latest. Now there is a security update and ubuntu:latest is updated in the docker repo .

1. How would I know my local image and its containers are running behind?

2. Is there some best practice for automatically updating local images and containers to follow the docker repo updates, which in practice would give you the same niceties of having unattended-upgrades running on a conventional ubuntu-machine

Answer 1

You can use Watchtower to watch for updates to the image a container is instantiated from and automatically pull the update and restart the container using the updated image. However, that doesn't solve the problem of rebuilding your own custom images when there's a change to the upstream image it's based on. You could view this as a two-part problem: (1) knowing when an upstream image has been updated, and (2) doing the actual image rebuild. (1) can be solved fairly easily, but (2) depends a lot on your local build environment/practices, so it's probably much harder to create a generalized solution for that.

If you're able to use Docker Hub's automated builds, the whole problem can be solved relatively cleanly using the repository links feature, which lets you trigger a rebuild automatically when a linked repository (probably an upstream one) is updated. You can also configure a webhook to notify you when an automated build occurs. If you want an email or SMS notification, you could connect the webhook to IFTTT Maker. I found the IFTTT user interface to be kind of confusing, but you would configure the Docker webhook to post to https://maker.ifttt.com/trigger/docker_xyz_image_built/with/key/your_key.

If you need to build locally, you can at least solve the problem of getting notifications when an upstream image is updated by creating a dummy repo in Docker Hub linked to your repo(s) of interest. The sole purpose of the dummy repo would be to trigger a webhook when it gets rebuilt (which implies one of its linked repos was updated). If you're able to receive this webhook, you could even use that to trigger a rebuild on your side.

Answer 2

Premise to my answer:

1. Containers are run with tags.
2. The same tag can be pointed to different image UUID as we please/ feel appropriate.
3. Updates done to an image can be committed to a new image layer

Approach

1. Build all the containers in the first place with a security-patch update script
2. Build an automated process for the following
   • Run an existing image to new container with security patch script as the command
   • Commit changes to the image as
     • existing tag -> followed by restarting the containers one by one
     • new version tag -> replace few containers with new tag -> validate -> move all containers to new tag

Additionally, the base image can be upgraded/ the container with a complete new base image can be built at regular intervals, as the maintainer feels necessary

Advantages

1. We are preserving the old version of the image while creating the new security patched image, hence we can rollback to previous running image if necessary
2. We are preserving the docker cache, hence less network transfer (only the changed layer gets on the wire)
3. The upgrade process can be validated in staging before moving to prod
4. This can be a controlled process, hence the security patches only when necessary/ deemed important can be pushed.

Answer 3

BLUF: finding the right insertion point for monitoring changes to a container is the challenge. It would be great if DockerHub would solve this. (Repository Links have been mentioned but note when setting them up on DockerHub - "Trigger a build in this repository whenever the base image is updated on Docker Hub. Only works for non-official images.")

While trying to solve this myself I saw several recommendations for webhooks so I wanted to elaborate on a couple of solutions I have used.

1. Use microbadger.com to track changes in a container and use it's notification webhook feature to trigger an action. I set this up with zapier.com (but you can use any customizable webhook service) to create a new issue in my github repository that uses Alpine as a base image.

   • Pros: You can review the changes reported by microbadger in github before taking action.
   • Cons: Microbadger doesn't let you track a specific tag. Looks like it only tracks 'latest'.

2. Track the RSS feed for git commits to an upstream container. ex. https://github.com/gliderlabs/docker-alpine/commits/rootfs/library-3.8/x86_64. I used zapier.com to monitor this feed and to trigger an automatic build of my container in Travis-CI anytime something is committed. This is a little extreme but you can change the trigger to do other things like open an issue in your git repository for manual intervention.

   • Pros: Closer to an automated pipline. The Travis-CI build just checks to see if your container has issues with whatever was committed to the base image repository. It's up to you if your CI service takes any further action.
   • Cons: Tracking the commit feed isn't perfect. Lots of things get committed to the repository that don't affect the build of the base image. Doesn't take in to account any issues with frequency/number of commits and any API throttling.

Answer 4

I'm not going into the whole question of whether or not you want unattended updates in production (I think not). I'm just leaving this here for reference in case anybody finds it useful. Update all your docker images to the latest version with the following command in your terminal:

# docker images | awk '(NR>1) && ($2!~/none/) {print $1":"$2}' | xargs -L1 docker pull

Answer 5

A 'docker way' would be to use docker hub automated builds. The Repository Links feature will rebuild your container when an upstream container is rebuilt, and the Webhooks feature will send you a notification.

It looks like the webhooks are limited to HTTP POST calls. You'd need to set up a service to catch them, or maybe use one of the POST to email services out there.

I haven't looked into it, but the new Docker Universal Control Plane might have a feature for detecting updated containers and re-deploying.

Answer 6

Above Answers are also correct

There are two Approach

1. Use webhooks
2. Run script for every specific minute to get fresh pull of docker images

I am just sharing script may be it will helpful for you! You can use it with cronjob, I tried succesfully on OSX

#!/bin/bash
##You can use below commented line for setting cron tab for running cron job and to store its O/P in one .txt file  
#* * * * * /usr/bin/sudo -u admin -i bash -c /Users/Swapnil/Documents/checkimg.sh > /Users/Swapnil/Documents/cron_output.log 2>&1
# Example for the Docker Hub V2 API
# Returns all images and tags associated with a Docker Hub organization account.
# Requires 'jq': https://stedolan.github.io/jq/

# set username, password, and organization
# Filepath where your docker-compose file is present
FILEPATH="/Users/Swapnil/Documents/lamp-alpine"
# Your Docker hub user name
UNAME="ur username"
# Your Docker hub user password
UPASS="ur pwd"
# e.g organisation_name/image_name:image_tag
ORG="ur org name"
IMGNAME="ur img name"
IMGTAG="ur img tag"
# Container name
CONTNAME="ur container name"
# Expected built mins
BUILDMINS="5"
#Generally cronjob frequency
CHECKTIME="5"
NETWORKNAME="${IMGNAME}_private-network"
#After Image pulling, need to bring up all docker services?
DO_DOCKER_COMPOSE_UP=true
# -------
echo "Eecuting Script @ date and time in YmdHMS: $(date +%Y%m%d%H%M%S)"
set -e
PIDFILE=/Users/Swapnil/Documents/$IMGNAME/forever.pid
if [ -f $PIDFILE ]
then
  PID=$(cat $PIDFILE)
  ps -p $PID > /dev/null 2>&1
  if [ $? -eq 0 ]
  then
    echo "Process already running"
    exit 1
  else
    ## Process not found assume not running
    echo $$
    echo $$ > $PIDFILE
    if [ $? -ne 0 ]
    then
      echo "Could not create PID file"
      exit 1
    fi
  fi
else
  echo $$ > $PIDFILE
  if [ $? -ne 0 ]
  then
    echo "Could not create PID file"
    exit 1
  fi
fi

# Check Docker is running or not; If not runing then exit
if docker info|grep Containers ; then
    echo "Docker is running"
else
    echo "Docker is not running"
    rm $PIDFILE
    exit 1
fi

# Check Container is running or not; and set variable
CONT_INFO=$(docker ps -f "name=$CONTNAME" --format "{{.Names}}")
if [ "$CONT_INFO" = "$CONTNAME" ]; then
    echo "Container is running"
    IS_CONTAINER_RUNNING=true
else
    echo "Container is not running"
    IS_CONTAINER_RUNNING=false
fi


# get token
echo "Retrieving token ..."
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${UNAME}'", "password": "'${UPASS}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)

# get list of repositories
echo "Retrieving repository list ..."
REPO_LIST=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/?page_size=100 | jq -r '.results|.[]|.name')

# output images & tags
echo "Images and tags for organization: ${ORG}"
echo
for i in ${REPO_LIST}
do
  echo "${i}:"
  # tags
  IMAGE_TAGS=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${i}/tags/?page_size=100 | jq -r '.results|.[]|.name')
  for j in ${IMAGE_TAGS}
  do
    echo "  - ${j}"
  done
  #echo
done

# Check Perticular image is the latest or not
#imm=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100)
echo "-----------------"
echo "Last built date details about Image ${IMGNAME} : ${IMGTAG} for organization: ${ORG}"
IMAGE_UPDATED_DATE=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100 | jq -r '.results|.[]|select(.name | contains("'${IMGTAG}'")).last_updated')
echo "On Docker Hub IMAGE_UPDATED_DATE---$IMAGE_UPDATED_DATE"
echo "-----------------"

IMAGE_CREATED_DATE=$(docker image inspect ${ORG}/${IMGNAME}:${IMGTAG} | jq -r '.[]|.Created')
echo "Locally IMAGE_CREATED_DATE---$IMAGE_CREATED_DATE"

updatedDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_UPDATED_DATE:0:16}" +%Y%m%d%H%M%S) 
createdDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_CREATED_DATE:0:16}" +%Y%m%d%H%M%S)
currentDate=$(date +%Y%m%d%H%M%S)

start_date=$(date -jf "%Y%m%d%H%M%S" "$currentDate" "+%s")
end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
updiffMins=$(( ($start_date - $end_date) / (60) ))
if [[ "$updiffMins" -lt $(($CHECKTIME+1)) ]]; then
        if [ ! -d "${FILEPATH}" ]; then
            mkdir "${FILEPATH}";
        fi
        cd "${FILEPATH}"
        pwd
        echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
        echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
        echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
        echo "Found after regular checking time -> Docker hub's latest updated image is new; Diff ${updiffMins} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Script is checking for latest updates after every ${CHECKTIME} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Fetching all new"
        echo "---------------------------"
        if $IS_CONTAINER_RUNNING ; then
            echo "Container is running"         
        else
            docker-compose down
            echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
        fi
        echo "Image_Created_Date=$currentDate" > ".env"
        echo "ORG=$ORG" >> ".env"
        echo "IMGNAME=$IMGNAME" >> ".env"
        echo "IMGTAG=$IMGTAG" >> ".env"
        echo "CONTNAME=$CONTNAME" >> ".env"
        echo "NETWORKNAME=$NETWORKNAME" >> ".env"
        docker-compose build --no-cache
        echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
        if $DO_DOCKER_COMPOSE_UP ; then
            docker-compose up -d
            echo "Docker services are up now, checked in" >> "ScriptOutput_${currentDate}.txt"  
        else
            echo "Docker services are down, checked in" >> "ScriptOutput_${currentDate}.txt"
        fi
elif [[ "$updatedDate" -gt "$createdDate" ]]; then 
    echo "Updated is latest"
    start_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
    end_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
    diffMins=$(( ($start_date - $end_date) / (60) ))
    if [[ "$BUILDMINS" -lt "$diffMins" ]]; then
        if [ ! -d "${FILEPATH}" ]; then
            mkdir "${FILEPATH}";
        fi
        cd "${FILEPATH}"
        pwd
        echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
        echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
        echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
        echo "Found after comparing times -> Docker hub's latest updated image is new; Diff ${diffMins} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Actual image built time is less i.e. ${diffMins} mins than MAX expexted BUILD TIME i.e. ${BUILDMINS} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Fetching all new" >> "ScriptOutput_${currentDate}.txt"
        echo "-----------------------------"
        if $IS_CONTAINER_RUNNING ; then
            echo "Container is running"         
        else
            docker-compose down
            echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
        fi
        echo "Image_Created_Date=$currentDate" > ".env"
        echo "ORG=$ORG" >> ".env"
        echo "IMGNAME=$IMGNAME" >> ".env"
        echo "IMGTAG=$IMGTAG" >> ".env"
        echo "CONTNAME=$CONTNAME" >> ".env"
        echo "NETWORKNAME=$NETWORKNAME" >> ".env"
        docker-compose build --no-cache
        echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
        if $DO_DOCKER_COMPOSE_UP ; then
            docker-compose up -d
            echo "Docker services are up now" >> "ScriptOutput_${currentDate}.txt"  
        else
            echo "Docker services are down" >> "ScriptOutput_${currentDate}.txt"
        fi
    elif [[ "$BUILDMINS" -gt "$diffMins" ]]; then
        echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
        echo "Docker images not fetched"
    else
        echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
        echo "Docker images not fetched"
    fi
elif [[ "$createdDate" -gt "$updatedDate" ]]; then 
    echo "Created is latest"
    start_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
    end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
    echo "Docker hub has older docker image than local; Older than $(( ($start_date - $end_date) / (60) ))mins"
fi
echo 
echo "------------end---------------"
rm $PIDFILE

Here is my docker-compose file

version:  "3.2"
services:
  lamp-alpine:
    build:
      context: .
    container_name: "${CONTNAME}"
    image: "${ORG}/${IMGNAME}:${IMGTAG}"
    ports:
      - "127.0.0.1:80:80"
    networks:
      - private-network 

networks:
  private-network:
    driver: bridge