Read stdin in chunks in Bash pipe

问题:

I have some shell scripts that works with pipes like such:

foo.sh | bar.sh

My bar.sh calls some command line program that can only take a certain number of lines of stdin. Thus, I want foo.sh's large stdout to be chunked up in N number of lines to make multiple bar.sh calls. Essentially, paginate foo.sh's stdout and do multiple bar.sh.

Is it possible? I am hoping for some magic in between the pipes like foo.sh | ??? | bar.sh. xargs -n doesn't quite get me what I want.

回答1:

I am nowhere near a machine to test this, but you need GNU Parallel to make this easy - along the lines of:

foo.sh | parallel --pipe -N 10000 -k bar.sh

As an added bonus, that will run as many bar.sh in parallel as you have CPU cores.

Add -j 1 if you only want one bar.sh at a time.

Add --dry-run if you want to see what it would do but without doinng anything.

回答2:

Use a while read loop.

foo.sh | while read line1 && read line2 && read line3; do
    printf "%s\n%s\n%s\n" "$line1" "$line2" "$line3" | bar.sh
done

For large N, write a function that loops.

read_n_lines() {
    read -r line || return 1
    echo "$line"
    n=$(($1 - 1))
    while [[ $n -gt 0 ]] && read -r line; do
        echo "$line"
        n=$((n-1))
    done
}

Then you can do:

n=20
foo.sh | while lines=$(read_n_lines $n); do
    printf "%s\n" "$lines" | bar.sh
done