Dealing with very long running rake task

Question

I am interested in running a very long running rake task, one that would take hours to complete and I am interested in learning about best practices for dealing with this problem.

Possible solutions I have found:

1. Set up a cron job
2. delayed_job
3. resque

cron seems like a simple solution to set up, but is it ideal for a very long task? What do you use and what are the advantages/disadvantages of your solution?

Answer 1

Personally I love Resque, you can use the resque-scheduler gem for dealing with long running or periodic tasks.

If you don't have to run your task very often, you can demonize the the rake task to make sure it keeps running if your SSH session dies or something.

Try something like this:

nohup rake my:task &

nohup will send the output to nohup.out in the directory you run the task in, and will also let leave your ssh session without the process dying, secondly the & will run it as a deamon.

Answer 2

In one application I created, users had the ability to upload PDFs which were thumbnailed upon uploading to create the preview images. As the PDFs could be extremely large, thumbnailing could take a while and had to run in the background. To do this, I used

• Paperclip for the uploading,
• the delayed_paperclip gem to hand off thumbnailing to a background process,
• Resque, backed by Redis, to handle the worker queue, and
• the God gem, to launch the Redis and Resque workers and monitor the whole shebang.

On the plus side, you get the nifty Resque GUI to view your workers in action, and you've got God there to watch for (and kill, and restart) runaway processes (which tend to occur quite a bit when you're processing PDFs in ImageMagick), making the whole thing much more stable and reliable.

On the minus side, it's a heck of a lot harder to set up than a cron job. But having cron run a long, memory-intensive process unmonitored seems like a recipe for disaster to me.

Hope that helps!