*FASTEST* directory listing

Question

I have massive directories, and I would like to read all the files as fast as I can. I mean, not DirectoryInfo.GetFiles fast, but 'get-clusters-from-disk-low-level' fast.

Of course, .NET 2.0, c#

Similar question was here, but this approach wasn't any good:

C# Directory listing massive directory

Someone suggested pInvoke on FindFirst/FindNext. Anybody tried that and is able to share results?

Answer 1

For the best performance, it is possible to P/Invoke NtQueryDirectoryFile, documented as ZwQueryDirectoryFile.

(That short of accessing the disk directly and reading the raw file system structures directly, which usually is not practical.)

Answer 2

Try using something like this DirectoryManager and refine it by your needs. Works faster than the .NET Framework GetDirectories() or GetFiles() because we ommitted there cross-platform checkings and adaptations.

Answer 3

For a "normal" approach, basically everything boils down to FindFirstFile/FindNextFile, you don't really get much faster than that... and that isn't super-turbo-fast.

If you really need speed, look into reading the MFT manually - but know that this requires admin privileges, and is prone to break whenever NTFS gets updated (and, oh yeah, won't work for non-NTFS filesystems). You might want to have a look at this code which has USN and MFT stuff.

However, perhaps there's a different solution. If your app is running constantly and needs to pick up changes, you can start off by doing one slow FindFirstFile/FindNextFile pass, and then use directory change notification support to be informed of updates... that works for limited users, and doesn't depend on filesystem structures.