Compare two files line by line and generate the di

Question

I want to compare file1 with file2 and generate a file3 which contains the lines in file1 which are not present in file2.

Answer 1

Try

sdiff file1 file2

It ususally works much better in most cases for me. You may want to sort files prior, if order of lines is not important (e.g. some text config files).

For example,

sdiff -w 185 file1.cfg file2.cfg

Answer 2

If you need to solve this with coreutils the accepted answer is good:

comm -23 <(sort file1) <(sort file2) > file3

You can also use sd (stream diff), which doesn't require sorting nor process substitution and supports infinite streams, like so:

cat file1 | sd 'cat file2' > file3

Probably not that much of a benefit on this example, but still consider it; in some cases you won't be able to use comm nor grep -F nor diff.

Here's a blogpost I wrote about diffing streams on the terminal, which introduces sd.

Answer 3

Consider this:
file a.txt:

abcd
efgh

file b.txt:

abcd

You can find the difference with:

diff -a --suppress-common-lines -y a.txt b.txt

The output will be:

efgh 

You can redirict the output in an output file (c.txt) using:

diff -a --suppress-common-lines -y a.txt b.txt > c.txt

This will answer your question:

"...which contains the lines in file1 which are not present in file2."

Answer 4

The Unix utility diff is meant for exactly this purpose.

$ diff -u file1 file2 > file3

See the manual and the Internet for options, different output formats, etc.

Answer 5

Many answers already, but none of them perfect IMHO. Thanatos' answer leaves some extra characters per line and Sorpigal's answer requires the files to be sorted or pre-sorted, which may not be adequate in all circumstances.

I think the best way of getting the lines that are different and nothing else (no extra chars, no re-ordering) is a combination of diff, grep, and awk (or similar).

If the lines do not contain any "<", a short one-liner can be:

diff urls.txt* | grep "<" | sed 's/< //g'

but that will remove every instance of "< " (less than, space) from the lines, which is not always OK (e.g. source code). The safest option is to use awk:

diff urls.txt* | grep "<" | awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}'

This one-liner diffs both files, then filters out the ed-style output of diff, then removes the trailing "<" that diff adds. This works even if the lines contains some "<" themselves.

Answer 6

Use the Diff utility and extract only the lines starting with < in the output