Looping over pairs of values in bash

Question

I have 10 text files and I want to paste each file with its pair, such that I have 5 total files.

I tried the following:

for i in 4_1 5_1 6_1 7_1 8_1
do
for j in 4_2 5_2 6_2 7_2 8_2
do
paste ${i}.txt ${j}.txt > ${i}.${j}.txt
done
done

However, this code combines every possible combination instead of just combining the matching pairs.

So I would like file 4_1.txt to be paired with 4_2.txt, 5_1.txt with 5_2.txt, etc.

Answer 1

You can use an associative array:

animals=(dog cat mouse)
declare -A size=(
  [dog]=big
  [cat]=medium
  [mouse]=small
)
declare -A sound=(
  [dog]=barks
  [cat]=purrs
  [mouse]=cheeps
)
for animal in "${animals[@]}"; do
  echo "$animal is ${size[$animal]} and it ${sound[$animal]}"
done

This allows you traversing pairs, triples, etc. Credits: the original idea is taken from @CharlesDuffy-s answer.

Answer 2

If you want to use one variable and perform and action with it, you just need to use one loop:

for file in 4 5 6 7 8
do
   paste "${file}_1" "${file}_2"
done

This will do

paste 4_1 4_2
paste 5_1 5_2
...

Answer 3

There is a common pattern where you have pairs of files, where one name of the pair can be easily derived from the other. If the file you know the name of is X and the other file is Y, you have the following common use cases.

• For renaming, Y is X with an extension removed and/or a date stamp added.
• For transcoding, Y is X with a different extension and perhaps a different directory.
• For many data analysis tasks, X and Y share some parts of the file name, but have different parameters or extensions.

All of these lend themselves to the same rough code skeleton.

for x in path/to/base*.ext; do
    dir=${x%/*}   # Trim trailing file name, keep dir
    base=${x##*/} # Trim any leading directory

    # In this case, $y has a different subdirectory and a different extension
    y=${dir%/to}/from/${base%.ext}.newext

    # Maybe check if y exists?  Or doesn't exist?
    if [ -e "$y" ]; then
        echo "$0: $y already exists -- skipping" >&2
        continue
    fi

    mv or ffmpeg or awk or whatever "$x" and "$y"
done

The key here is the observation that y can be derived from x with some simple variable substitutions. So you loop over the x values, and figure out the corresponding y value inside the loop.

Here, we have used the shell's built-in ${variable#prefix} and ${variable%suffix} operators to return the variable's value with any leading prefix or trailing suffix, respectively, trimmed off. (There is also ## and %% to match the longest, instead of the shortest, possible match. The expression after # or % is a regular shell glob pattern.) These should usually be all you need, although you frequently see sed or awk scripts even for this trivial job (where really you should usually try to avoid an external process), as well as of course for more demanding transformations.

If you need to loop over x files scattered across different directories, maybe the loop should start with something like

 find dir1 dir2 etc/and/so/forth -type f -name 'x-files*.ext' -print |
 while IFS='' read -r x; do
     :

A commonly seen problem in similar questions is answers which fail to quote $x and $y correctly. Generally, any variable containing a file name should always be in double quotes.

Where X and Y are unrelated, a common solution is to loop over a here document containing the mapping:

while read -r x y; do
    : stuff with "$x" and "$y"
done <<'____HERE'
    first_x_value  first_y_value
    another_x      corresponding_y
    random         surprise
____HERE

Answer 4

I agree with the answer currently proposed by fedorqui in the context of the question currently asked. The below is given only to provide some more general answers.

One more general approach (for bash 4.0 or newer) is to store your pairs in an associative array:

declare -A pairs=( [4_1]=4_2 [5_1]=5_2 [6_1]=6_2 [7_1]=7_2 [8_1]=8_2 )
for i in "${!pairs[@]}"; do
  j=${pairs[$i]}
  paste "$i.txt" "$j.txt" >"${i}.${j}.txt"
done

Another (compatible with older releases of bash) is to use more than one conventional array:

is=( 4_1 5_1 6_1 7_1 8_1 )
js=( 4_2 5_2 6_2 7_2 8_2 )
for idx in "${!is[@]}"; do
  i=${is[$idx]}
  j=${js[$idx]}
  paste "$i.txt" "$j.txt" >"$i.$j.txt"
done

Answer 5

the above did not work for me, but the following does read values in pairs from an ordered list

(can be more than pairs adding extra 'read-lines' :-)

while read x; do
  read y
  echo $x $y
done << '___HERE'
X1
Y1
X2
Y2
X3
Y3
___HERE

produces

X1 Y1
X2 Y2
X3 Y3