Airflow inside docker running a docker container

Question

I have airflow running on an EC2 instance, and I am scheduling some tasks that spin up a docker container. How do I do that? Do I need to install docker on my airflow container? And what is the next step after. I have a yaml file that I am using to spin up the container, and it is derived from the puckel/airflow Docker image

Answer 1

I got a simpler solution working which just requires a short Dockerfile to build a derived image:

FROM puckel/docker-airflow

USER root
RUN groupadd --gid 999 docker \
    && usermod -aG docker airflow
USER airflow

and then

docker build -t airflow_image .
docker run -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /usr/bin/docker:/bin/docker:ro \
    -v /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7:ro \
    -d airflow_image

Answer 2

Finally resolved

My EC2 setup is running unbuntu Xenial 16.04 and using a modified the puckel/airflow docker image that is running airflow

Things you will need to change in the Dockerfile

Add USER root at the top of the Dockerfile

USER root

mounting docker bin was not working for me, so I had to install the

docker binary in my docker container

Install Docker from Docker Inc. repositories.

RUN curl -sSL https://get.docker.com/ | sh

search for wrapdocker file on the internet. Copy it into scripts directory in the folder where the Dockerfile is located. This starts the docker daemon inside airflow docker

Install the magic wrapper

ADD ./script/wrapdocker /usr/local/bin/wrapdocker RUN chmod +x /usr/local/bin/wrapdocker

add airflow as a user to the docker group so the airflow can run docker jobs

RUN usermod -aG docker airflow

switch to airflow user

USER airflow

Docker compose file or command line arguments to docker run

Mount docker socket from docker airflow to the docker image just installed

- /var/run/docker.sock:/var/run/docker.sock

You should be good to go !

Answer 3

You can spin up docker containers from your airflow docker container by attaching volumes to your container.

Example:

docker run -v /var/run/docker.sock:/var/run/docker.sock:ro -v /path/to/bin/docker:/bin/docker:ro your_airflow_image

You may also need to attach some libraries required by docker. This depends on the system you are running Docker on. Just read the error messages you get when running a docker command inside the container, it will indicate you what you need to attach.

Your airflow container will then have full access to Docker running on the host. So if you launch docker containers, they will run on the host running the airflow container.