Running cron python jobs within docker

问题:

I would like to run a python cron job inside of a docker container in detached mode. My set-up is below:

My python script is test.py

  #!/usr/bin/env python
  import datetime
  print "Cron job has run at %s" %datetime.datetime.now()

My cron file is my-crontab

* * * * * /test.py > /dev/console

and my Dockerfile is

FROM ubuntu:latest
RUN apt-get update && apt-get install -y software-properties-common python-software-properties && apt-get update

RUN apt-get install -y python cron
ADD my-crontab /
ADD test.py /
RUN chmod a+x test.py

RUN crontab /my-crontab
ENTRYPOINT cron -f

What are the potential problems with this approach? Are there other approaches and what are their pros and cons?

回答1:

Several issues that I faced while trying to get a cron job running in a docker container were:

1. time in the docker container is in UTC not local time;
2. the docker environment is not passed to cron;
3. as Thomas noted, cron logging leaves a lot to be desired and accessing it through docker requires a docker-based solution.

There are cron-specific issues and are docker-specific issues in the list, but in any case they have to be addressed to get cron working.

To that end, my current working solution to the problem posed in the question is as follows:

Create a docker volume to which all scripts running under cron will write:

# Dockerfile for test-logs

# BUILD-USING:        docker build -t test-logs .
# RUN-USING:          docker run  -d -v /t-logs --name t-logs test-logs
# INSPECT-USING:      docker run -t -i  --volumes-from t-logs ubuntu:latest /bin/bash

FROM stackbrew/busybox:latest

# Create logs volume
VOLUME /var/log

CMD  ["true"]

The script that will run under cron is test.py:

#!/usr/bin/env python

# python script which needs an environment variable and runs as a cron job
import datetime
import os

test_environ = os.environ["TEST_ENV"]
print "Cron job has run at %s with environment variable '%s'" %(datetime.datetime.now(), test_environ)

In order to pass the environment variable to the script that I want to run under cron, follow Thomas' suggestion and put a crontab fragment for each script (or group of scripts) that has need of a docker environment variable in /etc/cron.d with a placeholder XXXXXXX which must be set.

# placed in /etc/cron.d 
# TEST_ENV is an docker environment variable that the script test.py need

TEST_ENV=XXXXXXX
#
* * * * * root python /test.py >> /var/log/test.log

Instead of calling cron directly, wrap cron in a python script that does does things: 1. reads the environment variable from the docker environment variable and sets the environment variable in a crontab fragment.

#!/usr/bin/env python

# run-cron.py
# sets environment variable crontab fragments and runs cron

import os
from subprocess import call
import fileinput

# read docker environment variables and set them in the appropriate crontab fragment
environment_variable = os.environ["TEST_ENV"]

for line in fileinput.input("/etc/cron.d/cron-python",inplace=1):
    print line.replace("XXXXXXX", environment_variable)

args = ["cron","-f", "-L 15"]
call(args)

The Dockerfile that for the container in which the cron jobs run is as follows:

# BUILD-USING:        docker build -t test-cron .
# RUN-USING docker run --detach=true --volumes-from t-logs --name t-cron test-cron

FROM debian:wheezy
#
# Set correct environment variables.
ENV HOME /root
ENV TEST_ENV test-value

RUN apt-get update && apt-get install -y software-properties-common python-software-properties && apt-get update

# Install Python Setuptools
RUN apt-get install -y python cron

RUN apt-get purge -y python-software-properties software-properties-common && apt-get clean -y && apt-get autoclean -y && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

ADD cron-python /etc/cron.d/
ADD test.py /
ADD run-cron.py /

RUN chmod a+x test.py run-cron.py

# Set the time zone to the local time zone
RUN echo "America/New_York" > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata

CMD ["/run-cron.py"]

Finally, create the containers and run them:

1. Create the log volume (test-logs) container: docker build -t test-logs .
2. Run log volume: docker run -d -v /t-logs --name t-logs test-logs
3. Create the cron container: docker build -t test-cron .
4. Run the cron container: docker run --detach=true --volumes-from t-logs --name t-cron test-cron
5. To inspect the log files of the scripts running under cron: docker run -t -i --volumes-from t-logs ubuntu:latest /bin/bash. The log files are in /var/log.

回答2:

Here is a complement on rosksw answer.

There is no need to do some string replacement in the crontab file in order to pass environment variables to the cron jobs.

It is simpler to store the environment variables in a file when running the contrainer, then load them from this file at each cron execution. I found the tip here.

In the dockerfile:

CMD mkdir -p /data/log && env > /root/env.txt && crond -n

In the crontab file:

* * * * * root env - `cat /root/env.txt` my-script.sh

回答3:

Adding crontab fragments in /etc/cron.d/ instead of using root's crontab might be preferable.

This would:

• Let you add additional cron jobs by adding them to that folder.
• Save you a few layers.
• Emulate how Debian distros do it for their own packages.

Observe that the format of those files is a bit different from a crontab entry. Here's a sample from the Debian php package:

# /etc/cron.d/php5: crontab fragment for php5
#  This purges session files older than X, where X is defined in seconds
#  as the largest value of session.gc_maxlifetime from all your php.ini
#  files, or 24 minutes if not defined.  See /usr/lib/php5/maxlifetime

# Look for and purge old sessions every 30 minutes
09,39 *     * * *     root   [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)

Overall, from experience, running cron in a container does work very well (besides cron logging leaving a lot to be desired).

回答4:

Here's an alternative solution.

in Dockerfile

ADD docker/cron/my-cron /etc/cron.d/my-cron
RUN chmod 0644 /etc/cron.d/my-cron

ADD docker/cron/entrypoint.sh /etc/entrypoint.sh

ENTRYPOINT ["/bin/sh", "/etc/entrypoint.sh"]

in entrypoint.sh

 #!/usr/bin/env bash
  printenv | cat - /etc/cron.d/my-cron > ~/my-cron.tmp \
    && mv ~/my-cron.tmp /etc/cron.d/my-cron

cron -f

回答5:

We are using below solution. It supports both docker logs functionality and ability to hang the cron process in the container on PID 1 (if you use tail -f workarounds provided above - if cron crashes, docker will not follow restart policy):

cron.sh:

#!/usr/bin/env bash

printenv | cat - /etc/cron.d/cron-jobs > ~/crontab.tmp \
    && mv ~/crontab.tmp /etc/cron.d/cron-jobs

chmod 644 /etc/cron.d/cron-jobs

tail -f /var/log/cron.log &

cron -f

Dockerfile:

RUN apt-get install --no-install-recommends -y -q cron 

ADD cron.sh /usr/bin/cron.sh
RUN chmod +x /usr/bin/cron.sh

ADD ./crontab /etc/cron.d/cron-jobs
RUN chmod 0644 /etc/cron.d/cron-jobs

RUN touch /var/log/cron.log

ENTRYPOINT ["/bin/sh", "/usr/bin/cron.sh"]

crontab:

* * * * * root <cmd> >> /var/log/cron.log 2>&1

And please don't forget to add the creepy new line in your crontab

回答6:

Single Container Method

You may run crond within the same container that is doing something closely related using a base image that handles PID 0 well, like phusion/baseimage.

Specialized Container Method

May be cleaner would be to have another Container linked to it that just runs crond. For example:

Dockerfile

 FROM busybox
 ADD crontab /var/spool/cron/crontabs/www-data
 CMD crond -f

crontab

 * * * * * echo $USER

Then run:

 $ docker build -t cron .
 $ docker run --rm --link something cron

Note: In this case it'll run the job as www-data. Cannot just mount the crontab file as volume because it needs to be owned by root with only write access for root, else crond will run nothing. Also you'll have to run crond as root.

回答7:

Another possibility is to use Crython. Crython allows you to regularly schedule a python function from within a single python script / process. It even understands cron syntax:

@crython.job(expr='0 0 0 * * 0 *')
def job():
    print "Hello world"

Using crython avoids the various headaches of running crond inside a docker container - your job is now a single process that wakes up when it needs to, which fits better into the docker execution model. But it has the downside of putting the scheduling inside your program, which isn't always desirable. Still, it might be handy in some use cases.

回答8:

Don't mix crond and your base image. Prefer to use a native solution for your language (schedule or crython as said by Anton), or decouple it. By decoupling it I mean, keep things separated, so you don't have to maintain an image just to be the fusion between python and crond.

You can use Tasker, a task runner that has cron (a scheduler) support, to solve it, if you want keep things decoupled.

Here an docker-compose.yml file, that will run some tasks for you

version: "2"

services:
    tasker:
        image: strm/tasker
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
        environment:
            configuration: |
                logging:
                    level:
                        ROOT: WARN
                        org.springframework.web: WARN
                        sh.strm: DEBUG
                schedule:
                    - every: minute
                      task: helloFromPython
                tasks:
                    docker:
                        - name: helloFromPython
                          image: python:3-slim
                          script:
                              - python -c 'print("Hello world from python")'

Just run docker-compose up, and see it working. Here is the Tasker repo with the full documentation:

http://github.com/opsxcq/tasker