Should log file streams be opened/closed on each w

Question

Should log classes open/close a log file stream on each write to the log file or should it keep the log file stream open throughout the application's lifetime until all logging is complete?

I'm asking in context of a desktop application. I have seen people do it both ways and was wondering which approach yields the best all-around results for a logger.

Answer 1

In general, as everyone else said, keep the file open for performance (open is a relatively slow operation). However, you need to think about what's going to happen if you keep the file open and people either remove the log file or truncate it. And that depends on the flags used at open time. (I'm addressing Unix - similar considerations probably apply to Windows, but I'll accept correction by those more knowledgeable than me).

If someone sees the log file grow to, say, 1 MiB and then removes it, the application will be none the wiser, and Unix will keep the log data safe until the log is closed by the application. What's more, the users will be confused because they probably created a new log file with the same name as the old and are puzzled about why the application 'stopped logging'. Of course, it didn't; it is just logging to the old file that no-one else can get at.

If someone notices that the log file has grown to, say, 1 MiB and then truncates it, the application will also be none the wiser. Depending on how the log file was opened, though, you might get weird results. If the file was not opened with O_APPEND (POSIX-speak), then the program will continue to write at its current offset in the log file, and the first 1 MiB of the file will appear as a stream of zero bytes — which is apt to confuse programs looking at the file.

How to avoid these problems?

• Open the log file with O_APPEND.
• Periodically use fstat() on the file descriptor and check whether st_nlink is zero.

If the link count goes to zero, somebody removed your log file. Time to close it, and reopen a new one. By comparison with stat() or open(), fstat() should be quick; it is basically copying information directly out of stuff that is already in memory, no name lookup needed. So, you should probably do that every time you are about to write.

Suggestions:

• Make sure there is a mechanism to tell the program to switch logs.
• Make sure you log the complete date and time in the messages.

I suffer from an application that puts out time and not date. Earlier today, I had a message file that had some entries from 17th August (one of the messages accidentally included the date in the message after the time), and then some entries from today, but I can only tell that because I created them. If I looked at the log file in a weeks time, I could not tell which day they were created (though I would know the time when they were created). That sort of thing is annoying.

You might also look at what systems such as Apache do — they have mechanisms for handling log files and there are tools for dealing with log rotation. Note: if the application does keep a single file open, does not use append mode, and does not plan for log rotation or size limits, then there's not much you can do about log files growing or having hunks of zeroes at the start — other than restarting the application periodically.

You should make sure that all writes to the log complete as soon as possible. If you use file descriptors, there is only kernel buffering in the way; this may well be acceptable, but consider the O_SYNC or O_DSYNC options to open(). If you use file stream I/O, ensure that each write is followed by fflush(). If you have a multi-threaded application, ensure that each write() contains a complete message; do not try to write parts of a message separately. With file stream I/O, you may need to use flockfile() and relatives to group operations together. With file descriptor I/O, you may be able to use dprintf() to do formatted I/O to a file descriptor (though it is not absolutely clear that dprintf() makes a single call to write()), or perhaps writev() to write separate segments of data in a single operation.

Incidentally, the disk blocks that 'contain' the zeroes are not actually allocated on disk. You can really screw up people's backup strategies by creating files which are a few GiB each, but all except the very last disk block contain just zeroes. Basically (error checking and file name generation omitted for conciseness):

int fd = open("/some/file", O_WRITE|O_CREATE|O_TRUNC, 0444);
lseek(fd, 1024L * 1024L * 1024L, 0);
write(fd, "hi", 2);
close(fd);

This occupies one disk block on the disk - but 1 GiB (and change) on (uncompressed) backup and 1 GB (and change) when restored. Anti-social, but possible.

Answer 2

Open and close. Can save you from a corrupt file in case of a system crash.

Answer 3

I would open and close on each write (or batch of writes). If doing this causes a performance problem in a desktop application, it's possible you're writing to the log file too often (although I'm sure there can be legitimate reasons for lots of writes).

Answer 4

It's generally better to keep them open.

If you're concerned about being able to read them from another process, you need make sure that the share mode you use to open/create them allows others to read them (but not write to them, obviously).

If you're worried about losing data in the event of a crash, you should periodically flush/commit their buffers.

Answer 5

I don't see any reason to close it.

On the other hand, closing and reopening takes a little extra time.

Answer 6

If you have frequent read/writes it is more efficient to keep the file open for the lifetime with a single open/close.

You might want to flush periodically or after each write though in case your application crashes you might not have all the data written to your file. Use fflush on Unix based systems and FlushFileBuffers on Windows.

If you are running on Windows as well you can use the CreateFile API with FILE_FLAG_NO_BUFFERING to go direct to file on each write.

It is also better to keep the file open for the lifetime, because each time you open/close you might have a failure if the file is in use. For example you might have a backup application that runs and open/closes your file as it's backing it up. And this might cause your program to not be able to access your own file. Ideally you would want to keep your file open always and specify sharing flags on Windows (FILE_SHARE_READ). On Unix based systems, sharing will be default.