how to clean up docker overlay directory?

Question

I'm running docker via CoreOS and AWS's ECS. I had a failing image that got restarted many times, and the containers are still around- they filled my drive partition. Specifically, /var/lib/docker/overlay/ contains a large number of files/directories.

I know that docker-cleanup-volumes is a thing, but it cleans the /volumes directory, not the /overlay directory.

docker ps -a shows over 250 start attempts on my bad docker container. They aren't running, though.

Aside from rm -rf /var/lib/docker/overlay/*, how can I/should I clean this up?

Answer 1

We just started having this problem, and btafarelo's answer got me part of the way, or at least made me feel better about removing the sha256 entries.

System info: ec2 instances running CoreOS 1.12 behind an ELB

• Drain the docker instance from the ELB

• Shutdown docker

  systemctl stop docker
  rm -rf /var/lib/docker/overlay/*
  

• Execute the results of the commands

  for d in $(find /var/lib/docker/image/overlay -type d -name '*sha256*'); do echo rm -rf $d/* ; done
  

• reboot (easiest way to bring everything back up)

This recovered about 25% of the disk after the services restarted with no ill side affects.

Answer 2

docker ps

• --quiet

• --all

• --filter status=exited

---

docker rm

• --force

---

docker images

• --quiet
• --all
• --filter dangling=true

---

docker rmi

• -- force

---

Your hacky way is fine.

docker rm `docker ps -a | grep Exited | awk '{print $1 }'`

My hacky way is

docker rm $(docker ps --all | awk '/ago/{print $1}')

A slightly cleaner way is to run docker ps with the --quiet (-q) flag to get just the id number and --filter status=exited to --filter just the exited ones.

docker rm $(docker ps --filter status=exited --quiet) # remove stopped docker processes

or to run docker rm with the --force (-f) flag and docker ps with the --all (-a) flag to shut down even the running ones

docker rm --force $(docker ps --all --quiet) # remove all docker processes

---

What's probably taking up all that disk space after several failed builds is the images. To conserve disk space on the docker host, periodically remove unused docker images with

docker rmi $(docker images --filter dangling=true --quiet) # clean dangling docker images

or to get more aggressive, you can --force (-f) it to clean up --all (-a) images

docker rmi --force $(docker images --all --quiet) # clean all possible docker images

---

@analytik 's way of putting it into a .bashrc function seems like a practical idea

function cleanup_docker() {
  docker rm --force $(docker ps --all --quiet) # remove all docker processes
  docker rmi $(docker images --filter dangling=true --quiet) # clean dangling docker images
}

and if you're in the habit of generating lots of docker images that you don't need, add it to .bash_logout

---

Answer 3

I have added this to bashrc in my dev environment, and gotten used to running it every day or so.

function cleanup_docker() {
  docker ps -f status=exited -q | xargs -r docker rm
  docker images -f dangling=true -q | xargs -r docker rmi
}

In some cases, the following script can free up more space, as it will try to remove all images, and just fail silently:

function cleanup_docker_aggressive() {
  for i in $(docker images --no-trunc -q | sort -u)
  do
    docker rmi $i 2> /dev/null
  done
}

Sadly, they're not significantly cleaner than your solution.

EDIT: Starting with Docker 1.13, you can use docker system:

docker system df    # to check what is using space
docker system prune # cleans up also networks, build cache, etc

EDIT: Starting with Docker 2017.09, you can also use container and image

docker container prune
docker image prune -a

the latter you can use with fancy filters like --filter "until=24h"

Answer 4

here is resolution to clean docker overlay directory from https://lebkowski.name/docker-volumes/

docker images --no-trunc | grep '<none>' | awk '{ print $3 }' | xargs -r docker rmi

docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v

for Docker < 1.9 :

find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d | grep -vFf <(docker ps -aq | xargs docker inspect | jq -r '.[]|.Mounts|.[]|.Name|select(.)')

Or for Docker >=1.9 :

docker volume ls -qf dangling=true | xargs -r docker volume rm

Answer 5

Here's the hacky way I'm doing this right now. I'm not going to accept it as an answer because I'm hoping there's a better way.

# delete old docker processes
docker rm `docker ps -a | grep Exited | awk '{print $1 }'`
  ignore_errors: true

# delete old images. will complain about still-in-use images.
docker rmi `docker images -aq`

Answer 6

From our side we used:

sudo docker system prune -a -f

Which save me 3Go !

We use also the famous commands :

sudo docker rm -v $(sudo docker ps -a -q -f status=exited)
sudo docker rmi -f  $(sudo docker images -f "dangling=true" -q)
docker volume ls -qf dangling=true | xargs -r docker volume rm

We put that on cron to manage a little bit more efficently our disk space.

Cheers

Reference: https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604/4