piping output of process to sh while loop?

Question

I'm trying to loop over the output of a Perl process, line by line, within a while loop. However, I'm having difficulty with the syntax.

I tried this, but got an "ambiguous redirect" error:

#!/bin/sh

while read line; do 
    echo "$line" 
    # do stuff with $line 
done < $(perl process.pl)

./script.sh : line 6: $(perl process.pl): ambiguous redirect

For instance, one (inefficient) solution would be:

#!/bin/sh

tmpFile=/tmp/tmpFile.txt
perl process.pl > $tmpFile 
while read line; do 
    echo "$line" 
    # do stuff with $line 
done < $tmpFile

I know I can pipe the Perl process to a while loop as:

perl process.pl | while ...

but the while loop is executed in a subshell, and I need some variables set in the while loop to remain after the loop finished, so this is not an option.

What can I do?

Answer 1

Use a here-document:

while IFS= read -r line; do
    echo "$line" 
    # do stuff with $line 
done <<EOF
$(perl process.pl)
EOF

Answer 2

You're almost there. Try this:

while read -r line; do 
    echo "$line" 
    # do stuff with $line 
done < <(perl process.pl)

The only difference is the < instead of the $.

$(cmd) is a command substitution, which expands to the output of the command within the parentheses. On the other hand, <(cmd) is a process substitution. Note that this is a Bash extension, so you should also change your shebang to be #!/bin/bash if you want to use this method.

Alternatively, if you are not using Bash, you can simply use a pipe instead:

perl process.pl | while read -r line; do 
    echo "$line" 
    # do stuff with $line 
done

As an aside, you almost always want to use the -r switch with read

Answer 3

Use a named pipe; bash process substitution is essentially syntactic sugar around this.

mkfifo output
perl process.pl > output &
while IFS= read -r line; do 
    echo "$line" 
    # do stuff with $line 
done < output
rm output

The named pipe does not require process.pl to complete before the while loop begins consuming the output.