Write to a file from multiple threads asynchronous

Question

Here is my situation. I would like to make writing to the file system as efficient as possible in my application. The app is multi-threaded and each thread can possibly write to the same file. Is there a way that I can write to the file asynchronously from each thread without having the writes in the different threads bang heads together, so to speak?

I'm using C# and .NET 3.5, and I do have the Reactive Extensions installed as well.

Answer 1

Have a look at Asynchronous I/O. This will free up the cpu to continue with other tasks.
Combine with ReaderWriterLock as @Jack B Nimble mentioned

If by

writing to the file system as efficient as possible

you mean making the actual file I/O as fast as possible you are going to have a hard time speeding it up much, disk is just physically slower. Maybe SSD's?

Answer 2

While thread based locks can solve this, there is a manner which works across threads, but is probably best used when you have multiple processes writing to the end of a single file.

To get this behavior across processes (or threads too), specify that you want atomic append writes to the operating system when the OS file handles are created. This is done by specifying O_APPEND under Posix(Linux,Unix), and FILE_APPEND_DATA under Windows.

In C# you don't call the OS 'open', or 'CreateFile' system calls directly, but there are ways to get this result.

I asked how to do this under Windows a while ago, and got two good answers here: How can I do an atomic write/append in C#, or how do I get files opened with the FILE_APPEND_DATA flag?

Basically, you can use FileStream() or PInvoke, I would suggest FileStream() over PInvoke for obvious reasons.

You can use constructor arguments to FileStream() to specify asynchronous file I/O in addition to the FileSystemRights.AppendData flag, which should give you both async I/O and atomic append writes to a file.

Warning: Some OSes have limits on the maximum number of bytes that can be atomically written this way, and exceeding that threshold will remove the OS promise of atomicity.

Because of this last gotcha, I would recommend staying with lock() style contention management when trying to address your problem within a single process.

Answer 3

What I would do is have separate worker thread(s) dedicated to the task of writing files out. When one of your other threads needs to write some data out, it should call a function to add the data to an ArrayList (or some other container/class). Inside this function, there should be a lock statement at the top to prevent more than one thread from executing simultaneously. After adding the reference to the ArrayList it returns and continues on with its chores. There are a couple of ways to handle the writing thread(s). Probably the simplest is to simply put it into an infinite loop with a sleep statement at the end so that it does not chew up your cpu(s). Another way is to use thread primitives and go into a wait state when there is no more data to be written out. This method implies that you would have to activate the thread with something like the ManualResetEvent.Set method.

There are many different ways to read in and write out files in .NET. I have written a benchmark program and give the results in my blog:

http://designingefficientsoftware.wordpress.com/2011/03/03/efficient-file-io-from-csharp

I would recommend using the Windows ReadFile and WriteFile methods if you need performance. Avoid any of the asynchronous methods since my benchmark results show that you get better performance with synchronous I/O methods.

Bob Bryan MCSD

Answer 4

Use Reader / Writer locks to access the file stream.

Answer 5

For those who prefer code, I am using following to do remote logging from web apps...

public static class LoggingExtensions
{
    static ReaderWriterLock locker = new ReaderWriterLock();
    public static void WriteDebug(this string text)
    {
        try
        {
            locker.AcquireWriterLock(int.MaxValue); //You might wanna change timeout value 
            System.IO.File.AppendAllLines(Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Replace("file:\\", ""), "debug.txt"), new[] { text });
        }
        finally
        {
            locker.ReleaseWriterLock();
        }
    }
}

Hope this saves you some time