How can I get unique values from an array in Bash?

Question

I've got almost the same question as here.

I have an array which contains aa ab aa ac aa ad, etc. Now I want to select all unique elements from this array. Thought, this would be simple with sort | uniq or with sort -u as they mentioned in that other question, but nothing changed in the array... The code is:

echo `echo "${ids[@]}" | sort | uniq`

What am I doing wrong?

Answer 1

'sort' can be used to order the output of a for-loop:

for i in ${ids[@]}; do echo $i; done | sort

and eliminate duplicates with "-u":

for i in ${ids[@]}; do echo $i; done | sort -u

Finally you can just overwrite your array with the unique elements:

ids=( `for i in ${ids[@]}; do echo $i; done | sort -u` )

Answer 2

this one will also preserve order:

echo ${ARRAY[@]} | tr [:space:] '\n' | awk '!a[$0]++'

and to modify the original array with the unique values:

ARRAY=($(echo ${ARRAY[@]} | tr [:space:] '\n' | awk '!a[$0]++'))

Answer 3

Without loosing the original ordering:

uniques=($(tr ' ' '\n' <<<"${original[@]}" | awk '!u[$0]++' | tr '\n' ' '))

Answer 4

If your array elements have white space or any other shell special character (and can you be sure they don't?) then to capture those first of all (and you should just always do this) express your array in double quotes! e.g. "${a[@]}". Bash will literally interpret this as "each array element in a separate argument". Within bash this simply always works, always.

Then, to get a sorted (and unique) array, we have to convert it to a format sort understands and be able to convert it back into bash array elements. This is the best I've come up with:

eval a=($(printf "%q\n" "${a[@]}" | sort -u))

Unfortunately, this fails in the special case of the empty array, turning the empty array into an array of 1 empty element (because printf had 0 arguments but still prints as though it had one empty argument - see explanation). So you have to catch that in an if or something.

Explanation: The %q format for printf "shell escapes" the printed argument, in just such a way as bash can recover in something like eval! Because each element is printed shell escaped on it's own line, the only separator between elements is the newline, and the array assignment takes each line as an element, parsing the escaped values into literal text.

e.g.

> a=("foo bar" baz)
> printf "%q\n" "${a[@]}"
'foo bar'
baz
> printf "%q\n"
''

The eval is necessary to strip the escaping off each value going back into the array.

Answer 5

cat number.txt

1 2 3 4 4 3 2 5 6

print line into column: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}'

1
2
3
4
4
3
2
5
6

find the duplicate records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}' |awk 'x[$0]++'

4
3
2

Replace duplicate records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}' |awk '!x[$0]++'

1
2
3
4
5
6

Find only Uniq records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i|"sort|uniq -u"}

1
5
6

Answer 6

A bit hacky, but this should do it:

echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '

To save the sorted unique results back into an array, do Array assignment:

sorted_unique_ids=($(echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

If your shell supports herestrings (bash should), you can spare an echo process by altering it to:

tr ' ' '\n' <<< "${ids[@]}" | sort -u | tr '\n' ' '

Input:

ids=(aa ab aa ac aa ad)

Output:

aa ab ac ad

Explanation:

• "${ids[@]}" - Syntax for working with shell arrays, whether used as part of echo or a herestring. The @ part means "all elements in the array"
• tr ' ' '\n' - Convert all spaces to newlines. Because your array is seen by shell as elements on a single line, separated by spaces; and because sort expects input to be on separate lines.
• sort -u - sort and retain only unique elements
• tr '\n' ' ' - convert the newlines we added in earlier back to spaces.
• $(...) - Command Subsitution
• Aside: tr ' ' '\n' <<< "${ids[@]}" is a more efficient way of doing: echo "${ids[@]}" | tr ' ' '\n'