Docker Registry 2.0 - how to delete unused images?

Question

We updated our private docker registry to the official Registry 2.0. This version can now delete docker images identified by a hashtag (see https://docs.docker.com/registry/spec/api/#deleting-an-image) but I still don't see a way to cleanup old images.

As our CI server is continously producing new images, I would need a method to delete all images from the private registry which are no longer identified by a named tag.

If there's no built-in way to achieve this, I think a custom script could possibly work, but I don't see a v2 API method either to list all stored hashtags of an image..

How can I keep my private registry clean? Any hints?

Answer 1

I was looking for the same functionality in the registry v2 api but only found soft deleting which is not what I was looking for. While researching I found the Github project delete-docker-registry-image which removes the actual files from the mounted volume via a bash script. Not tested it maybe useful...

Answer 2

Deletion of images (you can keep 10 last versions, like I do in my CI) is done in three steps:

1. Enable image deletion by setting environment variable REGISTRY_STORAGE_DELETE_ENABLED: "true" and passing it to docker-registry

2. Run below script (it will delete all images and tags but keep last 10 versions)

   registry.py -l user:pass -r https://example.com:5000 --delete --num 10

3. Run garbage collection (you can put it into your daily cron task)

   docker-compose -f [path_to_your_docker_compose_file] run registry bin/registry garbage-collect /etc/docker/registry/config.yml

registry.py can be downloaded from the link below, it also allows listing images, tags and layers, as well as deleting a particular image and/or tag.

https://github.com/andrey-pohilko/registry-cli

Before garbage collection my registry folder was 7 Gb, after I ran the above steps it deflated down to 1 Gb.

Answer 3

Regarding your question:

I would need a method to delete all images from the private registry which are no longer identified by a named tag

A new version of the docker registry in distribution/registry:master has this nice feature! However, you won't be able to trigger it from the API.

Anyway, you will be able to clean all untagged manifests in your registry, meaning that every overwritten tag won't leave old manifests and blobs in the registry. Every "unused" layer will be cleaned by the Registry Garbage Collectior.

You just have to run a docker exec:

docker exec ${container_id} registry garbage-collect \ 
  /path/to/your/registry/config.yml \
  --delete-untagged=true

Looking at this garbage-collect binary help:

Usage: 
  registry garbage-collect <config> [flags]
Flags:
  -m, --delete-untagged=false: delete manifests that are not currently referenced via tag
  -d, --dry-run=false: do everything except remove the blobs
  -h, --help=false: help for garbage-collect

You can have a look at the github PR. It's been merged and usable with distribution/registry, tag master as of 2018-02-23. It supersedes the docker/docker-registry project with a new API design, focused around security and performance...

I did use this feature today and recovered 89% of registry space (5.7 GB vs. 55 GB). Then I switched back to stable registry.

Answer 4

I pieced together various parts of this thread and created an easy to use cleanup script in bash You can check it out in this gist cleanup.sh

Answer 5

This is doable, although ugly. You need to be running (I think) registry 2.3 or greater, and have enabled deleting (REGISTRY_STORAGE_DELETE_ENABLED=True env var or equivalent). The example commands below assume a local filestore in /srv/docker-registry, but I'd be surprised if something equivalent couldn't be cooked up for other storage backends.

For each repository you wish to tidy up, you need to enumerate the digest references that are no longer required. The easiest way to do this is per-tag, using latest as an example in this case, you'd do something like:

ls /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/index/latest/index/sha256 -t | tail -n +3

This will list all but the three most recent digests pushed to the latest tag. Alternately, if you don't care about tags so much, but just want to keep the last few references pushed, you can do:

ls /srv/docker-registry/v2/repositories/<repo>/_manifests/revisions/sha256 -t | tail -n +3

Then, you just delete the references you don't need:

for hash in ls /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/index/latest/index/sha256 -t | tail -n +3); do curl -X DELETE https://registry:5000/v2/<repo>/manifests/sha256:$hash; done

Finally, you need to do a GC run, because the registry implements "soft deletes", which doesn't actually delete anything, it just makes it unavailable:

docker exec docker-registry /bin/registry garbage-collect /path/to/config.yml

Yes, this is all messy as hell, grovelling around in the backend storage, because there's no API method to enumerate all digests associated with a given tag, but that's the way the cookie crumbles.

Answer 6

There is some discussion happening to design this - right now, there is no layer cleanup tool / endpoint.

I would encourage you to go to:

• https://github.com/docker/distribution/issues/106
• https://github.com/docker/distribution/issues/210

and/or reach out on Freenode IRC on #docker-distribution for more.