How to make sure data integrity after sync/fsync/s

Question

Based on sync manual page, there is no guarantee the disc will flush its cache after calling sync: "According to the standard specification (e.g., POSIX.1-2001), sync() schedules the writes, but may return before the actual writing is done. However, since version 1.3.20 Linux does actually wait. (This still does not guarantee data integrity: modern disks have large caches.) "

And, in fsync manual, there is no mention about this.

Is there ways to makes sure all writes to disc especially portable device (USB) has been finished after calling sync? I have encountered cases that data and metadata information has not fully written to disc after calling sync/fsync. I am curious how "Safely remove device" in windows/linux knows that all data has been fully written by the device itself.

Answer 1

I am curious how "Safely remove device" in windows/linux knows that all data has been fully written by the device itself.

For IXish systems:

Unmount the USB-device's partitions using the umount command or the umount() system call.

---

Doing

blockdev --flushbufs

might flush the device's buffer, but does not keep anybody from accessing the devices again and refill buffers.

---

Also there is this kernel interface in the /proc file system:

/proc/sys/vm/drop_caches

which can be used to flush different buffers:

Verbatim from https://www.kernel.org/doc/Documentation/sysctl/vm.txt

[...]

To free dentries and inodes:

  echo 2 > /proc/sys/vm/drop_caches

[...]

Answer 2

At least in principle, this is a Linux bug. The specification for sync functions is that the data is fully written to permanent storage; leaving it in a hardware cache is not conforming.

I'm not sure what the correct workaround is, but you can probably strace the hwparm utility running with the -F option (I think that's the right one) to see what it's doing (or read the source, but strace is a lot easier).