Shell pipe: Exit immediately when one command fail

问题:

I'm using a pipe of several commands in bash. Is there a way of configuring bash to terminate all commands in the whole pipeline immediately should one of the commands fail?

In my case, the first command, say command1, runs for a while until it produces some output. You might substitute command1 by (sleep 5 && echo "Hello"), for instance.

Now, command1 | false does fail after 5 seconds but not immediately.

This behavior seems to have something to do with the amount of output the command produces. For instance, find / | false returns immediately.

In general, I wonder why bash behaves like this. Can anyone imagine any situation where it is useful that code like command1 | non-existing-command does not exit at once?

PS: Using temporary files is not an option for me, as the intermediate results I pipe around are to big to be stored.

PPS: Neither set -e nor set -o pipefail seem to influence this phenomenon.

回答1:

The bash documentation says in its section about pipelines:

Each command in a pipeline is executed in its own subshell [...]

"In its own subshell" means that a new bash process is spawned, which then gets to execute the actual command. Each subshell starts successfully, even when it immediately determines that the command it is asked to execute doesn't exist.

This explains why the entire pipe can be set up successfully even when one of the commands is nonsense. Bash does not check if each command can be run, it delegates that to the subshells. That also explains why, for example, the command nonexisting-command | touch hello will throw a "command not found" error, but the file hello will be created nonetheless.

In the same section, it also says:

The shell waits for all commands in the pipeline to terminate before returning a value.

In sleep 5 | nonexisting-command, as A.H. pointed out, the sleep 5 terminates after 5 seconds, not immediately, hence the shell will also wait 5 seconds.

I don't know why the implementation was done this way. In cases like yours, the behavior is surely not as one would expect.

Anyway, one slightly ugly workaround is to use FIFOs:

mkfifo myfifo
./long-running-script.sh > myfifo &
whoops-a-typo < myfifo

Here, the long-running-script.sh is started and then the scripts fails immediately on the next line. Using mutiple FIFOs, this could be extended to pipes with more than two commands.

回答2:

sleep 5 doesn't produce any output until it finishes, while find / immediately produces output that bash attempts to pipe to false.

回答3:

The first program does not know whether the second is terminated or not until it tries to write some date into the pipe. In case the second is terminated, the first receives the SIGPIPE which usually causes immediate exit.

You can force the first line of output to be piped immediately after staring, like this:

(sleep 0.1; echo; command1) | command2

This 100ms sleep is intended to wait until possible command2 exit right after starting. Of course, if command2 exits after 2 seconds, and command1 will be silent for 60 seconds, the whole shell command will return only after 60.1 seconds.

回答4:

find / |false fails faster because the first write(2) system call from find fails with the error EPIPE (Broken pipe). This is because false has been already terminated and hence the pipe between these two commands has been closed already on one side.

If find would ignore that error (it could do so in theory) it would by also "fail slow".

(sleep 5 && echo "Hello") | false is "fail slow", because the first part, sleep, does not "test" the pipe by writing to it. After 5 seconds the echo also get the EPIPE error. Whether this error terminates the first part in this case or not is not important to the question.