Is rename() without fsync() safe?

Question

Is it safe to call rename(tmppath, path) without calling fsync(tmppath_fd) first?

I want the path to always point to a complete file. I care mainly about Ext4. Is the rename() promised to be safe in all future Linux kernel versions?

A usage example in Python:

def store_atomically(path, data):
    tmppath = path + ".tmp"
    output = open(tmppath, "wb")
    output.write(data)

    output.flush()
    os.fsync(output.fileno())  # The needed fsync().
    output.close()
    os.rename(tmppath, path)

Answer 1

If you only care about ext4 and not ext3 then I'd recommend using fsync on the new file before doing the rename. The fsync performance on ext4 seems to be much better than on ext3 without the very long delays. Or it might be the fact that writeback is the default mode (at least on my Linux system).

If you only care that the file is complete and not which file is named in the directory then you only need to fsync the new file. There's no need to fsync the directory too since it will point to either the new file with its complete data, or the old file.

Answer 2

No.

Look at libeatmydata, and this presentation:

Eat My Data: How Everybody Gets File IO Wrong

http://www.oscon.com/oscon2008/public/schedule/detail/3172

by Stewart Smith from MySql.

In case it is offline/no longer available, I keep a copy of it:

• The video here
• The presentation slides (online version of slides)

Answer 3

From ext4 documentation:

When mounting an ext4 filesystem, the following option are accepted:
(*) == default

auto_da_alloc(*)    Many broken applications don't use fsync() when 
noauto_da_alloc     replacing existing files via patterns such as
                    fd = open("foo.new")/write(fd,..)/close(fd)/
                    rename("foo.new", "foo"), or worse yet,
                    fd = open("foo", O_TRUNC)/write(fd,..)/close(fd).
                    If auto_da_alloc is enabled, ext4 will detect
                    the replace-via-rename and replace-via-truncate
                    patterns and force that any delayed allocation
                    blocks are allocated such that at the next
                    journal commit, in the default data=ordered
                    mode, the data blocks of the new file are forced
                    to disk before the rename() operation is
                    committed.  This provides roughly the same level
                    of guarantees as ext3, and avoids the
                    "zero-length" problem that can happen when a
                    system crashes before the delayed allocation
                    blocks are forced to disk.

Judging by the wording "broken applications", it is definitely considered bad practice by the ext4 developers, but in practice it is so widely used approach that it was patched in ext4 itself.

So if your usage fits the pattern, you should be safe.

If not, I suggest you to investigate further instead of inserting fsync here and there just to be safe. That might not be such a good idea since fsync can be a major performance hit on ext3 (read).

On the other hand, flushing before rename is the correct way to do the replacement on non-journaling file systems. Maybe that's why ext4 at first expected this behavior from programs, the auto_da_alloc option was added later as a fix. Also this ext3 patch for the writeback (non-journaling) mode tries to help the careless programs by flushing asynchronously on rename to lower the chance of data loss.

You can read more about the ext4 problem here.