Consistent reading and writing a file in python

Question

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values. I use the ConfigParser module to both read and write the file.

I am wondering if this scenario is capable of leading into an inconsistent state (like in middle of reading the settings file, the other script begins writing). I am unaware if there are any mechanism behind the scene to automatically protect against this situations.

If such inconsistencies are possible, what could I use to synchronize both scripts and mantain the integrity of the operations ?

Answer 1

When you write the config file write it to a temporary file first. When it's done, rename it to the correct name. The rename operation (os.rename) is normally implemented as an atomic operation on Unix systems, Linux and Windows, too, I think, so there will be no risk of the other process trying to read the config while the writing has not been finished yet.

Answer 2

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values.

There may be a race condition when the reader reads while the writer writes to the file, so that the reader may read the file while it is incomplete.

You can protect from this race by locking the file while reading and writing (see Linux flock() or Python lockfile module), so that the reader never observes the file incomplete.

Or, better, you can first write into a temporary file and when done rename it to the final name atomically. This way the reader and writer never block:

def write_config(config, filename):
    tmp_filename = filename + "~"
    with open(tmp_filename, 'wb') as file:
        config.write(file)
    os.rename(tmp_filename, filename)

When the writer uses the above method no changes are required to the reader.

Answer 3

There are al least two ways to address this issue (assuming you are on a unix-ish system):

• If you want to write, write to a temporary file first, then do something unix can do atomically, especially rename the temporary file into place.

• Lock the file during any operation, e.g. with the help of this filelock module.

Personally, I like the first option because it utilizes the OS, although some systems have had problems with the atomicity: On how rename is broken in Mac OS X - another limitation: the rename system call can not rename files across devices.