How to make sure an application keeps running on L

Question

I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.

The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.

I know I could cron-script ps through grep:

ps -A | grep appname

But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?

In case it makes any difference, it's Ubuntu.

Answer 1

I couldn't get Chris Wendt solution to work for some reason, and it was hard to debug. This one is pretty much the same but easier to debug, excludes bash from the pattern matching. To debug just run: bash ./root/makerun-mysql.sh. In the following example with mysql-server just replace the value of the variables for process and makerun for your process.

• Create a BASH-script like this (nano /root/makerun-mysql.sh):

#!/bin/bash
process="mysql"
makerun="/etc/init.d/mysql restart"
if ps ax | grep -v grep | grep -v bash | grep --quiet $process
then
    printf "Process '%s' is running.\n" "$process"
    exit
else
    printf "Starting process '%s' with command '%s'.\n" "$process" "$makerun"
    $makerun
fi
exit

• Make sure it's executable by adding proper file permissions (i.e. chmod 700 /root/makerun-mysql.sh)

• Then add this to your crontab (crontab -e):

# Keep processes running every 5 minutes
*/5 * * * * bash /root/makerun-mysql.sh

Answer 2

A nice, simple way to do this is as follows:

1. Write your server to die if it can't listen on the port it expects
2. Set a cronjob to try to launch your server every minute

If it isn't running it'll start, and if it is running it won't. In any case, your server will always be up.

Answer 3

Monit is perfect for this :)

You can write simple config files which tell monit to watch e.g. a TCP port, a PID file etc

monit will run a command you specify when the process it is monitoring is unavailable/using too much memory/is pegging the CPU for too long/etc. It will also pop out an email alert telling you what happened and whether it could do anything about it.

We use it to keep a load of our websites running while giving us early warning when something's going wrong.

-- Your faithful employee, Monit

Answer 4

You could make it a service launched from inittab (although some Linuxes have moved on to something newer in /etc/event.d). These built in systems make sure your service keeps running without writing your own scripts or installing something new.

Answer 5

I think a better solution is if you test the function, too. For example, if you had to test an apache, it is not enough only to test, if "apache" processes on the systems exist.

If you want to test if apache OK is, then try to download a simple web page, and test if your unique code is in the output.

If not, kill the apache with -9 and then do a restart. And send a mail to the root (which is a forwarded mail address to the roots of the company/server/project).

Answer 6

Since you're using Ubuntu, you may be interested in Upstart, which has replaced the traditional sysV init. One key feature is that it can restart a service if it dies unexpectedly. Fedora has moved to upstart, and Debian is in experimental, so it may be worth looking into.

This may be overkill for this situation though, as a cron script will take 2 minutes to implement.

#!/bin/bash
if [[ ! `pidof -s yourapp` ]]; then
    invoke-rc.d yourapp start
fi