Removing duplicate blocks of lines from a file

Question

I have a certain file structure like this

>ID1
data about ID1....
................
................

>ID2
data about ID2....
................
................
................
................
>ID3
data about ID3....
................
................
...............

>ID1
data about ID1....
................
>ID5
data about ID5....
................
................

I want to remove these duplicate blocks of IDs. For eg in the above case it is ID1. It should be noted that only the ID part is same, the data after that could be different. However, I want to keep the first one and remove all the other ones. How can I do this in shell scripting manner?

Answer 1

In awk

awk '/^>/{p=!($0 in a);a[$0]}p' file1