Log4Net and logging from parallel instances

Question

I'm using log4net in my project and there is one problem. The major function of the program takes a long time and I use logging to save information about it. I use FileAppender for saving log to file.

My application is on the shared(local) folder and there could be several instances of the application running from one path. In this case I could log information only from the first program, other instances of my applications couldn't log info because log file is locked.

When I'm using "log4net.Appender.FileAppender+MinimalLock" option there are cases of information loss. Not all logs from both instances are saved to file.

How can I solve this problem and log info from parallel instances? Also what about performance degradation when I use "MinimalLock" option?

Thanks. Hope for your help.

Answer 1

You can create a custom appender that opens the file to write and then closes it. Should it hit a locked file, it could pause and retry a small number of times.

In the custom appender you could also open the file in shared write mode that would allow multiple writers, but this won't prevent pieces of log lines from being merged together.

If you aren't writing a lot of data, the open/close mechanism, listed above, is probably your best option. Note, because of the constant opening and closing of the file, you could see a noticeable performance impact if you are logging a lot of data.

A more complicated mechanism, but one that could provide a high performance logging path: Write a logging service that receives log lines via TCP or UDP. The service would be responsible for buffering data and writing it to disk. We've used this approach in the past (not via Log4Net, but as a general solution) to improve log writing efficiency.

Answer 2

Definitely consider creating different log files for each process, perhaps with unique filenames generated using a timestamp.

Answer 3

Simply include the process id of the application in the log file name. Different instances of your app will then log to different files. Here is an example:

<appender name="MyRollingFileAppender" type="log4net.Appender.RollingFileAppender">
  <file type="log4net.Util.PatternString">
    <conversionPattern value="log_%processid.log" />
  </file>
<!-- ... -->

Answer 4

Perhaps you log to different files from each instance? Otherwise, you'd probably need to set up a separate process that is dedicated to logging. Each instance of your program would send it's log messages there and it would take care of appending it to the file. This could be accomplished using SocketAppender, perhaps. Also, I find that breaking up log output into chucks using RollingFileAppender is much easier to deal with.

Answer 5

Instead of MinimalLock using InterProcessLock can reduce the data loss incase of multiple processes accessing single logfile.

log4net.Appender.FileAppender+InterProcessLock

Answer 6

I think it's a typical situation when centralized logging solution is desirable. Instead of worrying about files and suffer from performance bottlenecks, I'd rather pump log statements asynchronously to some remote service which will take care of storing and handling the logs. Have a look at this log aggregator called logFaces, it was designed with the purpose of decoupling applications from managing their logs. It should work with standard log4net UDP appender and will partition your log data per applications, hosts, threads, etc.. while letting you create log files any time when they really needed.

Disclosure: I am the author of this product.