Fastest reading binary file reading and writing

Question

I am writing an application to read and parse files which may be 1 KB to 200 MB in size.

I have to parse it two times...

1. Extract an image contained in the file.

2. Parse that image To extract the contents of the image.

I generally use the file stream, buffered stream, binary reader and binary writer to read and write the contents.

Now, I want to know the fastest and most efficient way to read the file and extract the contents...

Is there a good method or a good class library?

NOTE: Unsafe code is OK!

Answer 1

The fastest and simplest way to read the file is simply:

var file = File.ReadAllBytes(fileName);

That will read the entire file as a byte array into memory. You can then go through it looking for what you need at memory array access speed (which is to say, extremely fast). This will almost certainly be faster than trying to process the file as you read it.

However, if this file will not comfortably fit in memory (and 81 MB will), then you will need to do this in chunks. If this is not needed, we can safely avoid that tricky discussion. The solutions in this case will be either:

1. If using .NET 4.0, use memory mapped files (more in What are the advantages of memory-mapped files?).

2. If not, you'll need to chunk read, cache and keep around what you think you'll need in memory (for efficiency) or re-reading it you simply can't keep it in memory. This can become messy and slow.