Inter-process communication without FIFOs

Question

Inside a BASH script we can have multiple processes running in background which intercommunicate using named pipes, FIFOs registered on the filesystem. An example of this could be:

#!/bin/bash
mkfifo FIFO

# BG process 1
while :; do echo x; done & >FIFO

# BG process 2
while :; do read; done & <FIFO

exit

I wonder if it's possible to do the same intercommunication between background processes of a script without using a FIFO on filesystem, maybe with some kind of file-descriptor redirection.

Answer 1

Have you considered the use of signals? If the only thing you need is to trigger an event (without passing arguments), using kill and trap works perfectly (be careful of the semantics though, use SIGUSR1 for instance).

You might need to rework the logic though, as in the example below:

subprocess_finished()
{
    np=$( jobs -p | wc -l )
}

start_processing()
{
    myfile="$1"
    # DO SOMETHING HERE!!
    kill -SIGUSR1 $2
}

CPUS=$( lscpu | grep "^CPU(s):" | rev | cut -f 1 -d ' ' | rev )
POLLPERIOD=5  # 5s between each poll
np=0
trap subprocess_finished SIGUSR1

for myfile in *
do 
        start_processing "$myfile" $$ &
        np=$( jobs -p | wc -l )
        echo "$( date +'%Y-%m-%d %H:%M:%S' ) [$!] Starting #$np on $CPUS: $myfile"

        if [ $np -eq $CPUS ] 
        then
            # Wait for one CPU to be free
            trap subprocess_finished SIGUSR1
            while [ $np -eq $CPUS ]
            do
                sleep $POLLPERIOD
            done
        fi
    done
done

# wait for the last subprocesses
while [ ! -z "$( jobs -rp )" ]
do
    sleep $POLLPERIOD
done

Answer 2

You could use nc (aka netcat) which allows connecting a script's standard streams to a network socket. Of course it also works on localhost, so you can use it for IPC between scripts. The bonus is the possibility to have scripts running on different hosts, which is not possible with FIFOs (OK, maybe on NFS it is, but that would be rather cumbersome to set up unless you already have the NFS in place).

Answer 3

I just want to point out that ugly hacks didn't wish to be born that way.

Part which receives data:

node -e "require('net').createServer(function(s){s.pipe(process.stdout)}).listen(1337)"

Part which sends data:

echo "write clean code they said" > /dev/tcp/localhost/1337
echo "it will pay off they said" > /dev/tcp/localhost/1337

---

Works even in MSysGit's Bash for Windows, to my surprise.

Answer 4

Here's an example that runs two subprocesses implemented as functions of the same shell-script... One subprocess generates numbers 1...5 (sleeps in between prints), the second one reads from a fixed filedescriptor (5, to which STDOUT of the first FD is redirected to), multiplies by 2 and prints again. The main process redirects STDOUT of that second process to another fixed filedescriptor (6) and later on reads from that one in the loop.

It works basically the same as you'd do in C-code with fd pairs created by the pipe(2) system call. To understand what's happening run the script under strace -f!

Bash Version is 4.2.24(1) running on Ubuntu/x86.

[ubuntu /home/chris]
$ bash --version
GNU bash, version 4.2.24(1)-release (i686-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Output of script:

[ubuntu /home/chris]
$ ./read_from_fd.sh
Got number 2.
Got number 4.
Got number 6.
Got number 8.
Got number 10.

Source code:

#!/bin/bash

# Generate data, output to STDOUT.
generate_five_numbers() {
        for n in `seq 5` ; do
                echo $n
                sleep 2
        done
}

# Read data from FD#5, multiply by two, output to STDOUT.
multiply_number_from_fd5_by_two() {
        while read n <&5 ; do
                echo "$(( $n * 2 ))"
        done
}

# choose your FD number wisely ;-)

# run generator with its output dup'ed to FD #5
exec 5< <( generate_five_numbers )

# run multiplyier (reading from fd 5) with output dup'ed to FD #6
exec 6< <( multiply_number_from_fd5_by_two )

# read numbers from fd 6
while read n <&6 ; do
        echo "Got number $n."
done

Process tree while running:

──read_from_fd.sh(8118)─┬─read_from_fd.sh(8119)───sleep(8123)
                        └─read_from_fd.sh(8120)

Answer 5

Bash 4 has coprocesses.

You can also use anonymous named pipes, aka process substitution in Bash 2, 3 or 4.