Log to memory and then write to file

Question

I want to know something, I have this loop that runs for all days (7 times) and then another loop inside it, that runs for all the records in the file. (about 100000), so all in all its about a 700000 times, now I want to log each processing in each loop, and log that to a file, say we are inside 1st loop for first time, and 2nd loop for first time, we log each time, what is done in a file. But the problem is that if I were to log each time, it would terribly hurt the performance, because of so many IO operations, what I was thinking is that is there any way, I could log each and every step to memory (memory stream or anything) and then at the end of outer loop, log all that memory stream data to file?

say if I have

for (int i=0; i<7; i++)
{
  for (int j=0; j<RecordsCount; j++)
  {
    SomeOperation();
    // I am logging to a file here right now
  }
}

Its hurting the performance terribly, if I remove the logging file, I could finish it a lot earlier. So I was thinking it would be better to log to memory first and write all that log from memory to file. So, is there any way to do this? If there are many, what's the best?

ps: here is a logger I found http://www.codeproject.com/Articles/23424/TracerX-Logger-and-Viewer-for-NET, but I can use any other custom logger or anything else if required.

EDIT : If I use memory stream, and then write it all to file, would that give me better performance then using File.AppendAllLines as suggested in answer by Yorye and zmbq, also if would that give me any performance gain over what Jeremy commented?

Answer 1

Why not use a proper logging framework instead of writing your own?

NLog for instance has buffering built in and it is easy to configure: https://github.com/nlog/NLog/wiki/BufferingWrapper-target

I suggest you focus on writing code that gives value to your project while reusing existing solutions for all the other stuff. That will probably make you more efficient and give better results :-)

Answer 2

This is just an example, but you can get the idea.

You really had the solution, you just needed to do it...

for (int i=0; i<7; i++)
{
    var entries = new List<string>();

    for (int j=0; j<RecordsCount; j++)
    {
        SomeOperation();

        // Log into list
        entries.Add("Operation #" + j + " results: " + bla bla bla);
    }

    // Log all entries to file at once.
    File.AppendAllLines("logFile.txt", entries);
}

Answer 3

Logging 700,000 lines to a file shouldn't take all time long as long as you adequate buffers. In fact, it shouldn't take longer if you do it inside the loop, compared to doing it at once outside the loop.

Don't use File.AppendAllLines or something similar, instead open a stream to the file, make sure you have a buffer in place, and write through it.