Docker in Docker cannot mount volume

问题:

I am running a Jenkins cluster where in the Master and Slave, both are running as a Docker containers.

The Host is latest boot2docker VM running on MacOS.

To allow Jenkins to be able to perform deployment using Docker, I have mounted the docker.sock and docker client from the host to the Jenkins container like this :-

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker -v $HOST_JENKINS_DATA_DIRECTORY/jenkins_data:/var/jenkins_home -v $HOST_SSH_KEYS_DIRECTORY/.ssh/:/var/jenkins_home/.ssh/ -p 8080:8080 jenkins

I am facing issues while mounting a volume to Docker containers that are run inside the Jenkins container. For example, if I need to run another Container inside the Jenkins container, I do the following :-

sudo docker run -v $JENKINS_CONTAINER/deploy.json:/root/deploy.json $CONTAINER_REPO/$CONTAINER_IMAGE 

The above runs the container, but the file "deploy.json" is NOT mounted as a file, but instead as a "Directory". Even if I mount a Directory as a Volume, I am unable to view the files in the resulting container.

Is this a problem, because of file permissions due to Docker in Docker case ?

Any pointers would be useful !

Thanks!

回答1:

A Docker container in a Docker container uses the parent HOST's Docker daemon and hence, any volumes that are mounted in the "docker-in-docker" case is still referenced from the HOST, and not from the Container.

Therefore, the actual path mounted from the Jenkins container "does not exist" in the HOST. Due to this, a new directory is created in the "docker-in-docker" container that is empty. Same thing applies when a directory is mounted to a new Docker container inside a Container.

Very basic and obvious thing which I missed, but realized as soon I typed the question.

回答2:

Regarding your use case related to Jenkins, you can simply fake the path by creating a symlink on the host:

ln -s $HOST_JENKINS_DATA_DIRECTORY/jenkins_data /var/jenkins_home

回答3:

Another way to go about this is to use either named volumes or data volume containers. This way, the container inside doesn't have to know anything about the host and both Jenkins container and the build container reference the data volume the same way.

I have tried doing something similar to what you are doing, except with an agent rather that using the Jenkins master. The problem was the same in that I couldn't mount the Jenkins workspace in the inner container. What worked for me was using the data volume container approach and the workspace files were visible to both the agent container and the inner container. What I liked about the approach is the both containers reference the data volume in the same way. Mounting directories with an inner container would be tricky as the inner container now needs to know something about the host that its parent container is running on.

I have detailed blog post about my approach here:

http://damnhandy.com/2016/03/06/creating-containerized-build-environments-with-the-jenkins-pipeline-plugin-and-docker-well-almost/

As well as code here:

https://github.com/damnhandy/jenkins-pipeline-docker

In my specific case, not everything is working the way I'd like it to in terms of the Jenkins Pipeline plugin. But it does address the issue of the inner container being able to access the Jenkins workspace directory.

回答4:

This also works via docker-compose and/or named volumes so you don't need to create a data only container, but you still need to have the empty directory on the host.

Host setup

Make host side directories and set permissions to allow Docker containers to access sudo mkdir -p /var/jenkins_home/{workspace,builds,jobs} && sudo chown -R 1000 /var/jenkins_home && sudo chmod -R a+rwx /var/jenkins_home

docker-compose.yml

version: '3.1'
services:
  jenkins:
    build: .
    image: jenkins
    ports:
      - 8080:8080
      - 50000:50000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - workspace:/var/jenkins_home/workspace/
      # Can also do builds/jobs/etc here and below
  jenkins-lts:
    build:
      context: .
      args:
        versiontag: lts
    image: jenkins:lts
    ports:
      - 8081:8080
      - 50001:50000
volumes:
  workspace:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /var/jenkins_home/workspace/

When you docker-compose up --build jenkins (you may want to incorporate this into a ready to run example like https://github.com/thbkrkr/jks where the .groovy scripts pre-configure Jenkins to be useful on startup) and then you will be able to have your jobs clone into the $JENKINS_HOME/workspace directory and shouldn't get errors about missing files/etc because the host and container paths will match, and then running further containers from within the Docker-in-Docker should work as well.

Dockerfile (for Jenkins with Docker in Docker)

ARG versiontag=latest
FROM jenkins/jenkins:${versiontag}

ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"

COPY jenkins_config/config.xml /usr/share/jenkins/ref/config.xml.override
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

USER root
RUN curl -L http://get.docker.io | bash && \
    usermod -aG docker jenkins
# Since the above takes a while make any other root changes below this line
# eg `RUN apt update && apt install -y curl`
# drop back to the regular jenkins user - good practice
USER jenkins
EXPOSE 8080

回答5:

A way to work around this issue is to mount a directory (inside your docker container in which you mounted your docker socket) using the exact same path for its destination. Then, when you run a container from within that container, you are able to mount anything within that mount's path into the new container using docker -v.

Take this example:

# Spin up your container from which you will use docker
docker run -v /some/dir:/some/dir -v /var/run/docker.sock:/var/run.docker.sock docker:latest

# Now spin up a container from within this container
docker run -v /some/dir:/usr/src/app $CONTAINER_IMAGE

The folder /some/dir is now mounted across your host, the intermediate container as well as your destination container. Since the mount's path exists on both the host as the "nearly docker-in-docker" container, you can use docker -v as expected.

It's kind of similar to the suggestion of creating a symlink on the host but I found this (at least in my case), a cleaner solution. Just don't forget to cleanup the dir on the host afterwards! ;)