I have an embedded system, on which I do telnet
and then I run an application in background:
./app_name &
Now if I close my terminal and do telnet
from other terminal and if I check then I can see this process is still running.
To check this I have written a small program:
#include<stdio.h>
main()
{
while(1);
}
I ran this program in my local linux pc in background and I closed the terminal.
Now, when I checked for this process from other terminal then I found that this process was also killed.
My question is:
- Why undefined behavior for same type of process?
- On which it is dependent?
- Is it dependent on version of Linux?
Who should kill jobs?
Normally, foreground and background jobs are killed by
SIGHUP
sent by kernel or shell in different circumstances.When does kernel send
SIGHUP
?Kernel sends
SIGHUP
to controlling process:Kernel sends
SIGHUP
to other process groups:Typically, the controlling process is your shell. So, to sum up:
SIGHUP
to the shell when real or pseudoterminal is disconnected/closed;SIGHUP
to foreground process group when the shell terminates;SIGHUP
to orphaned process group if it contains stopped processes.Note that kernel does not send
SIGHUP
to background process group if it contains no stopped processes.When does
bash
sendSIGHUP
?Bash sends
SIGHUP
to all jobs (foreground and background):SIGHUP
, and it is an interactive shell (and job control support is enabled at compile-time);huponexit
option is set (and job control support is enabled at compile-time).See more details here.
Notes:
bash
does not sendSIGHUP
to jobs removed from job list usingdisown
;nohup
ignoreSIGHUP
.More details here.
What about other shells?
Usually, shells propagate
SIGHUP
. GeneratingSIGHUP
at normal exit is less common.Telnet or SSH
Under telnet or SSH, the following should happen when connection is closed (e.g. when you close
telnet
window on PC):SIGHUP
tobash
;bash
receivesSIGHUP
, sendsSIGHUP
to all jobs and terminates;SIGHUP
and terminates.Problem
I can reproduce your issue using
bash
andtelnetd
frombusybox
ordropbear
SSH server: sometimes, background job doesn't receiveSIGHUP
(and doesn't terminate) when client connection is closed.It seems that a race condition occurs when server (
telnetd
ordropbear
) closes master side of pty:bash
receivesSIGHUP
and immediately kills background jobs (as expected) and terminates;bash
detectsEOF
on slave side of pty before handlingSIGHUP
.When
bash
detectsEOF
, it by default terminates immediately without sendingSIGHUP
. And background job remains running!Solution
It is possible to configure
bash
to sendSIGHUP
on normal exit (includingEOF
) too:Ensure that
bash
is started as login shell. Thehuponexit
works only for login shells, AFAIK.Login shell is enabled by
-l
option or leading hyphen inargv[0]
. You can configuretelnetd
to run/bin/bash -l
or better/bin/login
which invokes/bin/sh
in login shell mode.E.g.:
Enable
huponexit
option.E.g.:
Type this in
bash
session every time or add it to.bashrc
or/etc/profile
.Why does the race occur?
bash
unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.
You can start digging from
quit.h
in the source code.Thus, it seems that in our case
bash
sometimes receivesSIGHUP
when it's in a critical section.SIGHUP
handler execution is delayed, andbash
readsEOF
and terminates before exiting critical section or calling next interruption point.Reference
AFAIK in both cases the process should be killed. In order to avoid this you have to issue a nohup like the following:
This way your process will continue executing. Probably the telnet part is due to a BUG similar to this one:
https://bugzilla.redhat.com/show_bug.cgi?id=89653
When you close the terminal, shell sends
SIGHUP
to all background processes – and that kills them. This can be suppressed in several ways, most notably:nohup
When you run program with
nohup
it catchesSIGHUP
and redirect program output.disown
disown
tells shell not to sendSIGHUP
Is it dependent on version of linux?
It is dependent on your shell. Above applies at least for bash.
In order completely understand whats happening you need to get into
unix
internals a little bit.When you are running a command like this
./app_name &
The
app_name
is sent to background process group. You can check aboutunix
process groups hereWhen you close
bash
with normal exit it triggersSIGHUP
hangup signal to all its jobs. Some information onunix
job control is here.In order to keep your app running when you exit
bash
you need to make your app immune to hangup signal withnohup
utility.And finally this is how you need to do it.
nohup app_name & 2> /dev/null;