What is the fastest way for reading huge files in

Question

My program needs to read chunks from a huge binary file with random access. I have got a list of offsets and lengths which may have several thousand entries. The user selects an entry and the program seeks to the offset and reads length bytes.

The program internally uses a TMemoryStream to store and process the chunks read from the file. Reading the data is done via a TFileStream like this:

FileStream.Position := Offset;
MemoryStream.CopyFrom(FileStream, Size);

This works fine but unfortunately it becomes increasingly slower as the files get larger. The file size starts at a few megabytes but frequently reaches several tens of gigabytes. The chunks read are around 100 kbytes in size.

The file's content is only read by my program. It is the only program accessing the file at the time. Also the files are stored locally so this is not a network issue.

I am using Delphi 2007 on a Windows XP box.

What can I do to speed up this file access?

edit:

• The file access is slow for large files, regardless of which part of the file is being read.
• The program usually does not read the file sequentially. The order of the chunks is user driven and cannot be predicted.
• It is always slower to read a chunk from a large file than to read an equally large chunk from a small file.
• I am talking about the performance for reading a chunk from the file, not about the overall time it takes to process a whole file. The latter would obviously take longer for larger files, but that's not the issue here.

I need to apologize to everybody: After I implemented file access using a memory mapped file as suggested it turned out that it did not make much of a difference. But it also turned out after I added some more timing code that it is not the file access that slows down the program. The file access takes actually nearly constant time regardless of the file size. Some part of the user interface (which I have yet to identify) seems to have a performance problem with large amounts of data and somehow I failed to see the difference when I first timed the processes.

I am sorry for being sloppy in identifying the bottleneck.

Answer 1

The stock TMemoryStream in Delphi is slow due to the way it allocates memory. The NexusDB company has TnxMemoryStream which is much more efficient. There might be some free ones out there that work better.

The stock Delphi TFileStream is also not the most efficient component. Wayback in history Julian Bucknall published a component named BufferedFileStream in a magazine or somewhere that worked with file streams very efficiently.

Good luck.

Answer 2

If you open help topic for CreateFile() WinAPI function, you will find interesting flags there such as FILE_FLAG_NO_BUFFERING and FILE_FLAG_RANDOM_ACCESS . You can play with them to gain some performance.

Next, copying the file data, even 100Kb in size, is an extra step which slows down operations. It is a good idea to use CreateFileMapping and MapViewOfFile functions to get the ready for use pointer to the data. This way you avoid copying and also possibly get certain performance benefits (but you need to measure speed carefully).

Answer 3

Maybe you can take this approach:

Sort the entries on max fileposition and then to the following:

1. Take the entries that only need the first X MB of the file (till a certain fileposition)
2. Read X MB from the file into a buffer (TMemorystream
3. Now read the entries from the buffer (maybe multithreaded)
4. Repeat this for all the entries.

In short: cache a part of the file and read all entries that fit into it (multhithreaded), then cache the next part etc.

Maybe you can gain speed if you just take your original approach, but sort the entries on position.