Remove duplicate entries using a Bash script [dupl

Question

This question already has an answer here:

• How can I delete duplicate lines in a file in Unix? 8 answers

I want to remove duplicate entries from a text file, e.g:

kavitha= Tue Feb    20 14:00 19 IST 2012  (duplicate entry) 
sree=Tue Jan  20 14:05 19 IST 2012  
divya = Tue Jan  20 14:20 19 IST 2012  
anusha=Tue Jan 20 14:45 19 IST 2012 
kavitha= Tue Feb    20 14:00 19 IST 2012 (duplicate entry) 

Is there any possible way to remove the duplicate entries using a Bash script?

Desired output

kavitha= Tue Feb    20 14:00 19 IST 2012 
sree=Tue Jan  20 14:05 19 IST 2012  
divya = Tue Jan  20 14:20 19 IST 2012  
anusha=Tue Jan 20 14:45 19 IST 2012

Answer 1

It deletes duplicate, consecutive lines from a file (emulates "uniq").
First line in a set of duplicate lines is kept, rest are deleted.

sed '$!N; /^\(.*\)\n\1$/!P; D'

Answer 2

This might work for you:

cat -n file.txt |
sort -u -k2,7 |
sort -n |
sed 's/.*\t/    /;s/\([0-9]\{4\}\).*/\1/'

or this:

 awk '{line=substr($0,1,match($0,/[0-9][0-9][0-9][0-9]/)+3);sub(/^/,"    ",line);if(!dup[line]++)print line}' file.txt

Answer 3

You can sort then uniq:

$ sort -u input.txt

Or use awk:

$ awk '!a[$0]++' input.txt

Answer 4

Perl one-liner similar to @kev's awk solution:

perl -ne 'print if ! $a{$_}++' input

This variation removes trailing whitespace before comparing:

perl -lne 's/\s*$//; print if ! $a{$_}++' input

This variation edits the file in-place:

perl -i -ne 'print if ! $a{$_}++' input

This variation edits the file in-place, and makes a backup input.bak

perl -i.bak -ne 'print if ! $a{$_}++' input