How to make sure an application keeps running on L

Question

I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.

The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.

I know I could cron-script ps through grep:

ps -A | grep appname

But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?

In case it makes any difference, it's Ubuntu.

Answer 1

I have used a simple script with cron to make sure that the program is running. If it is not, then it will start it up. This may not be the perfect solution you are looking for, but it is simple and works rather well.

#!/bin/bash
#make-run.sh
#make sure a process is always running.

export DISPLAY=:0 #needed if you are running a simple gui app.

process=YourProcessName
makerun="/usr/bin/program"

if ps ax | grep -v grep | grep $process > /dev/null
then
    exit
else
    $makerun &
fi

exit

Then add a cron job every minute, or every 5 minutes.

Answer 2

I have used from cron "killall -0 programname || /etc/init.d/programname start". kill will error if the process doesn't exist. If it does exist, it'll deliver a null signal to the process (which the kernel will ignore and not bother passing on.)

This idiom is simple to remember (IMHO). Generally I use this while I'm still trying to discover why the service itself is failing. IMHO a program shouldn't just disappear unexpectedly :)

Answer 3

It's a job for a DMD (daemon monitoring daemon). there are a few around; but I usually just write a script that checks if the daemon is running, and run if not, and put it in cron to run every minute.

Answer 4

The supervise tool from daemontools would be my preference - but then everything Dan J Bernstein writes is my preference :)

http://cr.yp.to/daemontools/supervise.html

You have to create a particular directory structure for your application startup script, but it's very simple to use.

Answer 5

If you are using a systemd-based distro such as Fedora and recent Ubuntu releases, you can use systemd's "Restart" capability for services. It can be setup as a system service or as a user service if it needs to be managed by, and run as, a particular user, which is more likely the case in OP's particular situation.

The Restart option takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always.

To run it as a user, simply place a file like the following into ~/.config/systemd/user/something.service:

[Unit]
Description=Something

[Service]
ExecStart=/path/to/something
Restart=on-failure

[Install]
WantedBy=graphical.target

then:

systemctl --user daemon-reload
systemctl --user [status|start|stop|restart] something

No root privilege / modification of system files needed, no cron jobs needed, nothing to install, flexible as hell (see all the related service options in the documentation).

See also https://wiki.archlinux.org/index.php/Systemd/User for more information about using the per-user systemd instance.

Answer 6

Put your run in a loop- so when it exits, it runs again... while(true){ run my app.. }