Docker runs out of disk space even though containe

问题:

I have installed Docker Toolbox for Mac OSX and running several containers inside. First two I created were with Cassandra and were running fine. After that I've created 2 Debian containers, connected to bash through docker terminal with the purpose to install Oracle JDK8.

At the point when I was about to extract java from the tarball - I've got a ton of "Cannot write: No space left on device" error messages during the execution of "tar" command.

I've checked the space:

$ docker ps -s

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                         NAMES               SIZE

9d8029e21918        debian:latest       "/bin/bash"              54 minutes ago      Up 54 minutes                                                     deb-2               620.5 MB (virtual 744 MB)

49c7a0e37475        debian:latest       "/bin/bash"              55 minutes ago      Up 55 minutes                                                     deb-1               620 MB (virtual 743.5 MB)

66a17af83ca3        cassandra           "/docker-entrypoint.s"   4 hours ago         Up 4 hours          7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp   node-2              40.16 MB (virtual 412.6 MB)

After seeing that output I noticed that one of my nodes with cassandra is missing. In went to check to Kitematic and found out that it is in the DOWN state and I can't start it: "Cannot write node . No space left on device" - error message shown for this attempt.

Are there any limits that Docker has to run the containers?

When I remove all my cassandra ones and leave just a couple of Debian - java is able to be extracted from the tar. So the issue is definitely in some Docker settings related to sizing.

What is the correct way to resolve the issue with space limits here?

UPDATE.

$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE cassandra latest 13ea610e5c2b 11 hours ago 374.8 MB debian jessie 23cb15b0fcec 2 weeks ago 125.1 MB debian latest 23cb15b0fcec 2 weeks ago 125.1 MB

The output of df -hi

$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on none 251K 38K 214K 15% / tmpfs 251K 18 251K 1% /dev tmpfs 251K 12 251K 1% /sys/fs/cgroup tmpfs 251K 38K 214K 15% /etc/hosts shm 251K 1 251K 1% /dev/shm

`df -h 
Filesystem Size Used Avail Use% Mounted on 
none 1.8G 1.8G 0 100% 
/ tmpfs 1002M 0 1002M 0% 
/dev tmpfs 1002M 0 1002M 0% 
/sys/fs/cgroup tmpfs 1.8G 1.8G 0 100% 
/etc/hosts shm 64M 0 64M 0% /dev/shm`

Appreciate help.

回答1:

I have resolved this issue in docker somehow.

By default the memory for the docker is set to be 2048M by default.

First step I performed is stopping my docker machine:

$ docker-machine stop default

Then I went to the $HOME/.docker/machine/machines/default/config.json file and changed the "Memory" setting to be higher, i.e. 4096.

{
    "ConfigVersion": 3,
    "Driver": {
        "VBoxManager": {},
        "IPAddress": "192.168.99.102",
        "MachineName": "default",
        "SSHUser": "docker",
        "SSHPort": 59177,
        "SSHKeyPath": "/Users/lenok/.docker/machine/machines/default/id_rsa",
        "StorePath": "/Users/lenok/.docker/machine",
        "SwarmMaster": false,
        "SwarmHost": "tcp://0.0.0.0:3376",
        "SwarmDiscovery": "",
        "CPU": 1,
        "Memory": 4096, 
        "DiskSize": 204800,
        "Boot2DockerURL": "",
        "Boot2DockerImportVM": "",
        "HostDNSResolver": false,
        "HostOnlyCIDR": "192.168.99.1/24",
        "HostOnlyNicType": "82540EM",
        "HostOnlyPromiscMode": "deny",
        "NoShare": false,
        "DNSProxy": false
    },
    "DriverName": "virtualbox",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": [],
            "Dns": null,
            "GraphDir": "",
            "Env": [],
            "Ipv6": false,
            "InsecureRegistry": [],
            "Labels": [],
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": true,
            "RegistryMirror": [],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Master": false,
            "Host": "tcp://0.0.0.0:3376",
            "Image": "swarm:latest",
            "Strategy": "spread",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": [],
"config.json" [noeol] 75L, 2560C
            "Overcommit": 0,
            "ArbitraryFlags": [],
            "Env": null
        },
        "AuthOptions": {
            "CertDir": "/Users/lenok/.docker/machine/certs",
            "CaCertPath": "/Users/lenok/.docker/machine/certs/ca.pem",
            "CaPrivateKeyPath": "/Users/lenok/.docker/machine/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/Users/lenok/.docker/machine/machines/default/server.pem",
            "ServerKeyPath": "/Users/lenok/.docker/machine/machines/default/server-key.pem",
            "ClientKeyPath": "/Users/lenok/.docker/machine/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/Users/lenok/.docker/machine/certs/cert.pem",
            "ServerCertSANs": [],
            "StorePath": "/Users/lenok/.docker/machine/machines/default"
        }
    },
    "Name": "default" 
}

Finally, started my docker machine again:

$ docker-machine start default

回答2:

Issue 18869 refers to a docker-machine memory allocation problem.

This can be tested on the fly with

vboxmanage controlvm default 4096

Since drivers/virtualbox/virtualbox.go#L344-L352 reloads the settings from HOME/.docker/machine/machines/default/config.json, it is best to record that new value in that file (as mentioned in this answer).

That "No space left on device" was seen in docker/machine issue 2285, where the vmdk image created is a dynamically allocated/grow at run-time (default), creating a smaller on-disk foot-print initially, therefore even when creating a ~20GiB vm, with --virtualbox-disk-size 20000 requires on about ~200MiB of free space on-disk to start with.
And the default memory is quite low.

---

Make sure you that don't have :

• any more exited container that you could remove:

  docker rm -v $(docker ps --filter status=exited -q 2>/dev/null) 2>/dev/null
  

• any dangling images

  docker rmi $(docker images --filter dangling=true -q 2>/dev/null) 2>/dev/null
  

(Those are the result of rebuild which makes intermediate images unused)

See also "How to remove old and unused Docker images"

---

Then make sure you don't have an inode exhaustion problem, as in issue 10613.
Check df -hi (with i for inodes)

---

connected to bash through docker terminal with the purpose to install Oracle JDK8.

Try instead to specify the installation in a Dockerfile and build an image with the JDK installed.