I'm using this command to merge multiple identical directories and to remove duplicate lines from each of the corresponding files:
for f in app1/*; do
bn="$(basename "$f")"
sort -u "$f" "app2/$bn" > "app/$bn"
done
Is there a way to edit this so that it checks the lines of all the files and removes all the duplicates as well? I do need to keep the existing file structure with individual files.
The end result creates a directory with 300 text files that's no larger than 30mb.
Example:
**Directory app1**
*1.txt*
a
b
c
*2.txt*
d
e
f
**Directory app2**
*1.txt*
a
b
c
g
*2.txt*
a
b
c
d
e
f
**Results in Directory app**
*1.txt*
a
b
c
g
*2.txt*
a
b
c
d
e
f
Desired Result in Directory app Should Be:
*1.txt*
a
b
c
g
*2.txt*
d
e
f
As you can see it's not removing the duplicate "A B C" lines from 2.txt when it's also found in 1.txt. All lines in each file should remain unique and all duplicates should be removed.
