Parallel processes: appending outputs to an array

Question

I have a for loop in which a function task is called. Each call to the function returns a string that is appended to an array. I would like to parallelise this for loop. I tried using & but it does not seem to work.

Here is the code not parallelised.

task (){ sleep 1;echo "hello $1"; }
arr=()

for i in {1..3}; do
    arr+=("$(task $i)")
done

for i in "${arr[@]}"; do
    echo "$i x";
done

The output is:

hello 1 x
hello 2 x
hello 3 x

Great! But now, when I try to parallelise it with

[...]
for i in {1..3}; do
    arr+=("$(task $i)")&
done
wait
[...]

the output is empty.

Answer 1

Try encapsulating your adding step into a function or a temporary script, but send the items to a file instead. I think you will need some command that I can't remember on top of my head for dealing with file lock. See if you need to export things too. Then you source the file at the end. Something like:

echo 'arr=(' > temp
add() { echo item >> temp; }
...
export -f add
...
add &
...
echo ')' > temp
source temp
rm temp

Answer 2

Try encapsulating your adding step into a function or a temporary script. See if you need to export things too. Something like:

add() { arr+=(value); }
...
export -f add
...
add &

Answer 3

GNU Parallel is good at doing stuff in parallel :-)

task (){ sleep 1;echo "hello $1"; }

# Make "task" known to sub shells
export -f task

# Do tasks in parallel
parallel -k task ::: {1..3}

Sample Output

hello 1
hello 2
hello 3

I am suggesting you do - but Charles kindly points out that this is a known bash pitfall:

array=( $(parallel -k task ::: {1..3}) )

Charles' suggested solution is:

IFS=$'\n' read -r -d '' -a array < <(parallel -k task ::: 1 2 3 && printf '\0')

Answer 4

You are looking for parset (part of GNU Parallel since 20170422) or env_parset (available since 20171222):

# If you have not run:
#    env_parallel --install
# and logged in again, then you can instead run this to activate (env_)parset:
. `which env_parallel.bash`

task (){
  echo "hello $1"
  sleep 1.$1
  perl -e 'print "binary\001\002\n"'
  sleep 1.$1
  echo output of parallel jobs do not mix
}
env_parset arr task ::: {1..3}
env_parset a,b,c task ::: {1..3}

echo "${arr[1]}" | xxd
echo "$b" | xxd

parset is supported in Bash/Ksh/Zsh (including arrays), ash/dash (without arrays).