Docker Machine: No space left on device

Question

I'm trying to set up Docker Machine with Docker Compose.

Scenario 1 (without Docker Machine)
If I run docker-compose up -d without Docker Machine, it creates my 3 linked containers as intented (nginx+mongodb+nodejs).

Scenario 2 (with Docker Machine)
Then I create a VM using Docker Machine and tell Docker to talk to that machine with eval $(docker-machine env streambacker-dev).

At this point, if I ssh to my docker machine and run df -h, I get:

If I then run docker-compose up -d, I get a "no space left on device" error while downloading the last container.

"tmpfs" seems to be indeed a bit full after that:

Checking the --virtualbox-disk-size option shows that it defaults to 20000 MB, which I think is what we can see as "/dev/sda1" on both pictures. So why are containers filling up "tmpfs" n and what exactly is "tmpfs"? Is is a temporary download directory? How can I create more space for my containers?

Thanks!

For information, I'm using Docker Machine 0.4.0-rc2 and Docker Compose 1.3.2.

Answer 1

On docker osx / I was able to press a button [Move Disk Image] and it successfully moved the Docker.qcow2 (presumably containing containers / images)

initially - when machines started - I was still getting a No space left on device error but it resolved shortly after.

Answer 2

I ran into this problem and could not add additional space with the docker UI for mac, I installed docker with homebrew and ran the following command when creating my machine:

docker-machine create --driver virtualbox --virtualbox-memory "2048" --virtualbox-disk-size "40000" default

this adds double the space for memory and disk size to the virtualbox that I had before and you can add the settings size here that you need as you see fit

Answer 3

If you are using Docker Community Edition:

 docker system prune
 docker volume prune  # as suggested by @justin-m-chase since system prune does not clean volumes.

If you are using boot2docker (docker-machine) clear the volumes that are orphaned:

 docker volume rm $(docker volume ls -qf dangling=true)

Clear unused images:

 docker rmi $(docker images -q -f "dangling=true")

Answer 4

A. REMOVE UNUSED IMAGES

using the docker rm or docker rmi commands you can remove the images that you don't need. Actually exist an image that helps in this task (martin/docker-cleanup-volumes). The basis is to start selectig from your images and containers list:

docker ps -a -s

B. MODIFY THE DOCKER JSON DESCRIPTOR

it's mentioned in some forums. The idea is to increment the descriptor located in ~/.docker/machine/machines/default/config.json . The param seems to be DiskSize but i don't know if it works in other OSs (not in windows).

C. LINUX RESIZE:

in Windows OS, docker machine or boot2docker is in fact a virtualbox vm, then you can follow the procedure to resize the disk. Take care to backup the files. The general procedure is to make a resize in virtualbox and then use an utilitary called gpartd to modify the space perceived by linux in its partitions. There are some links to do this procedure referenced below:

• resize vbox disk
• move space
• vbox forum

D. RECREATE THE DOCKER-MACHINE / BOOT2DOCKER

The idea is recreate the default docker-machine. The following commands can illustrate you. Note that as you are re-creating the boot2docker, you will lost the previous downloaded docker images.

docker-machine rm default

docker-machine create --driver virtualbox --virtualbox-disk-size "100100" default

docker-machine env default

then you can go to virtual box and see the boot2docker space with the command "df -h"

Answer 5

I had the same error ([ERROR] InnoDB: Error number 28 means 'No space left on device') and solve it this way:

1 . Delete the orphaned volumes in Docker, you can use the built-in docker volume command. The built-in command also deletes any directory in /var/lib/docker/volumes that is not a volume so make sure you didn't put anything in there you want to save.

Warning be very careful with this if you have some data you want to keep

Cleanup:

$ docker volume rm $(docker volume ls -qf dangling=true)

Additional commands:

List dangling volumes:

$ docker volume ls -qf dangling=true

List all volumes:

$ docker volume ls

---

2 . Also consider removing all the unused Images.

First get rid of the <none> images (those are sometimes generated while building an image and if for any reason the image building was interrupted, they stay there).

here's a nice script I use to remove them

docker rmi $(docker images | grep "^<none>" | awk '{print $3}')

Then if you are using Docker Compose to build Images locally for every project. You will end up with a lot of images usually named like your folder (example if your project folder named Hello, you will find images name Hello_blablabla). so also consider removing all these images

you can edit the above script to remove them or remove them manually with

docker rmi {image-name}

Answer 6

Like said above, the tmpfs has nothing to do with --virtualbox-disk-size. It seems like boot2docker mounts tmpfs into memory, so you need to dedicate more memory to your virtualbox vm. You can do it by specifying the --virtualbox-memory parameter.

   --virtualbox-memory "1024"
Size of memory for host in MB [$VIRTUALBOX_MEMORY_SIZE]

Defaults:

$ docker-machine create --driver virtualbox testA
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testA
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testA:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    112.7M    783.9M  13% /

With --virtualbox-memory set to 8096

$ docker-machine create --driver virtualbox --virtualbox-memory 8096 testB
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testB
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testB:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     6.9G    112.4M      6.8G   2% /