C# - Hash contents of MS Office documents without

问题:

I am trying to identify files with duplicate contents. Decided to do a comparison using a hashing mechanism (MD5, SHA1 or any other). Works fine with ".txt" files. However, with MS Office files (.doc,.docx,.xls, etc) this fails.

MD5/SHA1 hash is not constant for MS Office files, even if they have the same "text" content. I assume MS Office stores some kind of meta-data in the file, which changes each time you save the file. Thus the hash is different.

e.g. I have a file ABC.doc and I compute the hash (Hash1) for it. Open and change 1 word and save the file. Undo the change you made and save and compute hash (Hash2). Hash1 != Hash2 in this case. It is same if you try this on a ".txt" file

Is there a way to de-dupe MS Office documents based on hashing its contents? Can we hash only the contents of a file and not its meta-data?

回答1:

I don't think this can be done without extracting the text of the document using a tool and then hashing the text. I can recommend Stellent Outside In, now owned by Oracle. But that could be an overkill solution to your needs. They provide a tool to extract text from many types of files, including all office files and versions.