How to automatically restart delayed_job when depl

Question

I'm hosting a rails project on Amazon Elastic Beanstalk and I try to configure a container command to automatically restart my delayed_job worker on the server after each deployment.

I tried with this one :

container_commands:
  restartdelayedjob:
    command: "RAILS_ENV=production script/delayed_job --pid-dir=/home/ec2-user/pids start"
    cwd: /var/app/current

But, it seems that the pushed version is deployed after the restarting of the worker so the jobs failed to be processed by the worker.

When I connect on my instance by ssh, kill the worker process and restart a new one from the deployed version folder, everything works fine.

Do you have any ideas of how I can handle this?

Thanks

Answer 1

As per the Amazon documentation for container_commands:

They run after the application and web server have been set up and the application version file has been extracted, but before the application version is deployed.

(emphasis mine)

This means at that point /var/app/current which you are setting as the cwd for your command is still pointing to the previous version. However by default, from the docs again, cwd:

is the directory of the unzipped application.

This means that if you want to run delayed_job from the directory of the app that just got extracted (but not yet deployed), don't override cwd and it should start the delayed_job for the app that's about to be deployed.

Ref: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#customize-containers-format-container_commands

Update:

I've now set this up myself and found there's limitations to doing it via the standard container_commands - basically delayed_job will be started while it is still in the /var/app/ondeck directory. Usually this is OK, but I had some issues with some jobs because that path had stuck around it would cause errors as the app was now in /var/app/current.

I found an undocumented (so warning!) approach that you can add scripts to be run AFTER your app server is restarted (and your new deploy is in /var/app/current).

Basically Elastic Beanstalk will execute any scripts in /opt/elasticbeanstalk/hooks/appdeploy/post after the web server is restarted. This means if you drop shell scripts in this directory they will be run.

I created a shell script like this:

#!/usr/bin/env bash
. /opt/elasticbeanstalk/support/envvars
cd $EB_CONFIG_APP_CURRENT
su -c "RAILS_ENV=production script/delayed_job --pid-dir=$EB_CONFIG_APP_SUPPORT/pids restart" $EB_CONFIG_APP_USER

I uploaded this script to an S3 bucket, and made sure it was "public". You can then use an options script in your .ebextensions directory (eg. 99delayed_job.config) to deploy this script as part of your app deploy, taking note that the post directory might not exist:

commands:
  create_post_dir:
    command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh":
    mode: "000755"
    owner: root
    group: root
    source: http://YOUR_BUCKET.s3.amazonaws.com/99_restart_delayed_job.sh

When you deploy you should see something like this in your /var/log/eb-tools.log:

2013-05-16 01:20:53,759 [INFO] (6467 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing directory: /opt/elasticbeanstalk/hooks/appdeploy/post/
2013-05-16 01:20:53,760 [INFO] (6467 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh
2013-05-16 01:21:02,619 [INFO] (6467 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Output from script: delayed_job: trying to stop process with pid 6139...
delayed_job: process with pid 6139 successfully stopped.

2013-05-16 01:21:02,620 [INFO] (6467 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Script succeeded.

As I said, putting stuff in this "post" directory is undocumented - but hopefully at some point Amazon add actual support to the .options scripts to run commands post-deploy, in that case you could move this to the officially supported approach.

Answer 2

On 64bit Amazon Linux 2014.09 v1.1.0 running Ruby 2.1 (Passenger Standalone), got it working thanks to this post.

Note that this script is run as root, but your workers should be run as the webapp user.

# Adds a post-deploy hook such that after a new version is deployed
# successfully, restarts the delayed_job workers.
#
# http://stackoverflow.com/questions/14401204/how-to-automatically-restart-delayed-job-when-deploying-a-rails-project-on-amazo
# http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_delayed_job.sh":
    mode: "000755"
    owner: root
    group: root
    encoding: plain
    content: |
      #!/usr/bin/env bash

      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
      EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      cd $EB_APP_CURRENT_DIR

      # Switch to the webapp user.  Worker shouldn't be run as root.
      su -s /bin/bash -c "bundle exec bin/delayed_job --pid-dir=$EB_APP_PIDS_DIR restart" $EB_APP_USER

Answer 3

In case anybody is looking to get delayed_job working in latest ElasticBeanstalk (64bit Amazon Linux 2014.09 v1.0.9 running Ruby 2.1 (Puma)): I got it to work using the below code (Thanks to damontorgerson). This file goes in ruby.config in .ebextensions folder.

# Install git in order to be able to bundle gems from git
packages:
yum:
  git: []

files: 
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_delayed_job":
    mode: "000777"
    owner: root
    group: root
    content: |
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_APP_STAGING_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_staging_dir)
      EB_CONFIG_APP_CURRENT=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_CONFIG_APP_LOGS=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
      EB_CONFIG_APP_PIDS=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      cd $EB_CONFIG_APP_CURRENT

      . $EB_SUPPORT_DIR/envvars.d/sysenv

      bin/delayed_job --pid-dir=/var/tmp restart

Answer 4

I got mine working like so with the "daemons" gem:

commands:
  create_post_dir:
    command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
  webapp_pids:
    command: "mkdir /home/webapp/pids"
    ignoreErrors: true
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars
      chown webapp:webapp /home/webapp/pids
      su -l -c "$EB_CONFIG_APP_CURRENT/bin/delayed_job --pid-dir=/home/webapp/pids restart" $EB_CONFIG_APP_USER
      echo "worker starting" >> /var/log/directory-hooks-executor.log