On Windows, how to open for writing a file already

Question

I'm trying to open a logfile which is kept open by another process and remove the first few lines. On Unix I'd simply do a os.open('/tmp/file.log', os.O_NONBLOCK) and that would get me closer to my goal.

Now i'm stuck with Windows and I need to rotate this log somehow without ending the application holding the file. Is this even possible?

At first I considered opening a file handle on the location where the application expected the log to be and just act as a pipe into a file-handle in Python but I couldn't find any way of doing that either on Windows.

I also thought of just moving the file on a regular basis and letting the application recreate the file but since it's being used by another process that doesn't do much good.

Thought of O_SHLOCK as well but then again, that's Unix and not Windows. So I went for mmap the file and hope that it would make it a bit more flexible but that led me nowhere.

import mmap
import contextlib
import time

with open(r'test.log', 'r+') as f:
    with contextlib.closing(mmap.mmap(f.fileno(), 0)) as m:
        while 1:
            line = m.readline()
            if len(line) > 0:
                print line
            time.sleep(0.5)

This results in that the application can't access the file because Python is holding it (and vice versa).

Came to think of signal.SIGHUP but that doesn't exist in Windows either so back to square one.

I'm stuck and I've tried it all, can Python help me here or do I need to switch my language?

Answer 1

Even if the application opens the file as a shared object Python can't so they can't get along by the looks of it.

It's not so bad :). You can (have to) open a file using CreateFile as pointed out by Augusto. You can use standard ctypes module for this. In the question Using a struct as a function argument with the python ctypes module you can see how to do it. Then you have to associate a C run-time file descriptor with an existing operating-system file handle you obtained in the previous step. You can use _open_osfhandle from the MS C run-time library (CRT) to do this. You can call it once again using ctypes; you can access it as ctypes.cdll.msvcrt._open_osfhandle. Then you have to associate Python file object with an existing C run-time file descriptor you obtained in the previous step. To do this in Python 3 you simply pass file descriptor as the first argument to the built-in open function. According to docs

file is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped.

In Python 2 you have to use os.fdopen; its task, according to docs, is to

Return an open file object connected to the file descriptor fd

All of the above should not be required to do such a simple thing. There's hope it will be much simpler when CPython's implementation on Windows starts using native Windows API for files instead of going through C run-time library which does not give access to many features of Windows platform. See Add new io.FileIO using the native Windows API issue for details.

Answer 2

Do you have any control over the application generating the logfile? Because depending on the way the file is open by that application, you really can't modify it.

This link may seem off-topic here, but deep in Windows, what determines the file access to other application is the dwShareMode parameter of the CreateFile function: http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx

The application should enable FILE_SHARE_WRITE and possibly FILE_SHARE_DELETE, plus it should flush and update the file position everytime it writes a file. Looking at the Python documentation for open(), there is no such detailed parameter.