How do I clear stuck/stale Resque workers?

Question

As you can see from the attached image, I've got a couple of workers that seem to be stuck. Those processes shouldn't take longer than a couple of seconds.

I'm not sure why they won't clear or how to manually remove them.

I'm on Heroku using Resque with Redis-to-Go and HireFire to automatically scale workers.

Answer 1

Adding to answer by hagope, I wanted to be able to only unregister workers that had been running for a certain amount of time. The code below will only unregister workers running for over 300 seconds (5 minutes).

Resque.workers.each {|w| w.unregister_worker if w.processing['run_at'] && Time.now - w.processing['run_at'].to_time > 300}

I have an ongoing collection of Resque related Rake tasks that I have also added this to: https://gist.github.com/ewherrmann/8809350

Answer 2

This avoids the problem as long as you have a resque version newer than 1.26.0:

resque: env QUEUE=foo TERM_CHILD=1 bundle exec rake resque:work

Keep in mind that it does not let the currently running job finish.

Answer 3

I had a similar problem that Redis saved the DB to disk that included invalid (non running) workers. Each time Redis/resque was started they appeared.

Fix this using:

Resque::Worker.working.each {|w| w.done_working}
Resque.redis.save # Save the DB to disk without ANY workers

Make sure you restart Redis and your Resque workers.

Answer 4

In your console:

queue_name = "process_numbers"
Resque.redis.del "queue:#{queue_name}"

Otherwise you can try to fake them as being done to remove them, with:

Resque::Worker.working.each {|w| w.done_working}

EDIT

A lot of people have been upvoting this answer and I feel that it's important that people try hagope's solution which unregisters workers off a queue, whereas the above code deletes queues. If you're happy to fake them, then cool.

Answer 5

you can also use below command to stop all rescue worker

sudo kill -9  `ps aux | grep resque | grep -v grep | cut -c 10-16`

reference from this link

Answer 6

Started working on https://github.com/shaiguitar/resque_stuck_queue/ recently. It's not a solution to how to fix stuck workers but it addresses the issue of resque hanging/being stuck, so I figured it could be helpful for people on this thread. From README:

"If resque doesn't run jobs within a certain timeframe, it will trigger a pre-defined handler of your choice. You can use this to send an email, pager duty, add more resque workers, restart resque, send you a txt...whatever suits you."

Been used in production and works pretty well for me thus far.