How to monitor delayed_job with monit

Question

Are there any examples on the web of how to monitor delayed_job with Monit?

Everything I can find uses God, but I refuse to use God since long running processes in Ruby generally suck. (The most current post in the God mailing list? God Memory Usage Grows Steadily.)

Update: delayed_job now comes with a sample monit config based on this question.

Answer 1

If your monit is running as root and you want to run delayed_job as my_user then do this:

/etc/init.d/delayed_job:

#!/bin/sh
#   chmod 755 /etc/init.d/delayed_job
#   chown root:root /etc/init.d/delayed_job

case "$1" in
  start|stop|restart)
    DJ_CMD=$1
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit
esac

su -c "cd /var/www/my_app/current && /usr/bin/env bin/delayed_job $DJ_CMD" - my_user

/var/www/my_app/shared/monit/delayed_job.monitrc:

check process delayed_job with pidfile /var/www/my_app/shared/tmp/pids/delayed_job.pid
start program = "/etc/init.d/delayed_job start"
stop  program = "/etc/init.d/delayed_job stop"
if 5 restarts within 5 cycles then timeout

/etc/monit/monitrc:

# add at bottom
include /var/www/my_app/shared/monit/*

Answer 2

I found it was easier to create an init script for delayed job. It is available here: http://gist.github.com/408929 or below:

#! /bin/sh
set_path="cd /home/rails/evatool_staging/current"

case "$1" in
  start)
        echo -n "Starting delayed_job: "
                su - rails -c "$set_path; RAILS_ENV=staging script/delayed_job start" >> /var/log/delayed_job.log 2>&1
        echo "done."
        ;;
  stop)
        echo -n "Stopping sphinx: "
                su - rails -c "$set_path; RAILS_ENV=staging script/delayed_job stop" >> /var/log/delayed_job.log 2>&1
        echo "done."
        ;;
      *)
            N=/etc/init.d/delayed_job_staging
            echo "Usage: $N {start|stop}" >&2
            exit 1
            ;;
    esac

    exit 0

Then make sure that monit is set to start / restart the app so in your monitrc file:

check process delayed_job with pidfile "/path_to_my_rails_app/shared/pids/delayed_job.pid"
start program = "/etc/init.d/delayed_job start"
stop program = "/etc/init.d/delayed_job stop"

and that works great!

Answer 3

to see what is going on, run monit in foreground verbose mode: sudo monit -Iv

using rvm installed under user "www1" and group "www1".

in file /etc/monit/monitrc:

#delayed_job
check process delayed_job with pidfile /home/www1/your_app/current/tmp/pids/delayed_job.pid
    start program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job start'" as uid www1 and gid www1
    stop program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job stop'" as uid www1 and gid www1
    if totalmem is greater than 200 MB for 2 cycles then alert

Answer 4

I don't know with Monit, but I've written a couple Munin plugins to monitor Queue Size and Average Job Run Time. The changes I made to delayed_job in that patch might also make it easier for you to write Monit plugins in case you stick with that.

Answer 5

I found a nice way to start delayed_job with cron on boot. I'm using whenever to control cron.

My schedule.rb:

# custom job type to control delayed_job
job_type :delayed_job, 'cd :path;RAILS_ENV=:environment script/delayed_job ":task"'

# delayed job start on boot
every :reboot do
  delayed_job "start"
end

Note: I upgraded whenever gem to 0.5.0 version to be able to use job_type

Answer 6

Since i didn't want to run as root, I ended up creating a bash init script that monit used for starting and stopping (PROGNAME would be the absolute path to script/delayed_job):

start() {
    echo "Starting $PROGNAME"
    sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME start
}

stop() {
    echo "Stopping $PROGNAME"
    sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME stop
}