comparing two text files and remove duplicates in

Question

I have two text files, file1 and file2.

File1 contains a bunch of random words, and file2 contains words that I want to remove from file1 when they occur. Is there a way of doing this?

I know I probably should include my own attempt at a script, to at least show effort, but to be honest it's laughable and wouldn't be of any help.

If someone could at least give a tip about where to start, it would be greatly appreciated.

Answer 1

get the words from each:

f1 = open("/path/to/file1", "r") 
f2 = open("/path/to/file2", "r") 

file1_raw = f1.read()
file2_raw = f1.read()

file1_words = file1_raw.split()
file2_words = file2_raw.split()

if you want unique words from file1 that aren't in file2:

result = set(file1_words).difference(set(file2_words))

if you care about removing the words from the text of file1

for w in file2_words:
    file1_raw = file1_raw.replace(w, "")

Answer 2

If you read the words into a set (one for each file), you can use set.difference(). This works if you don't care about the order of the output.

If you care about the order, read the first file into a list, the second into a set, and remove all the elements in the list that are in the set.

a = ["a", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog"]
b = {"quick", "brown"}
c = [x for x in a if not x in b]
print c

gives: ['a', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog']