rsync suddenly hanging indefinitely during transfe

Question

For the past few years, I have been using an rsync one-liner to back up important folders on my Mac Mini desktop (OSX 10.9, 2.5 GHz i5, 4 GB RAM) to a FreeNAS box (0.7.2 Sabanda revision 5266, Pentium D 2.66 GHz, 822MiB RAM [reported by the system, I think there's 1 GB in there]). I am running an rsync daemon on the FreeNAS box. Recently, these transfers have been hanging indefinitely. I have done the usual Google-fu and am unable to identify the source of the problem or a solution.

The one-liner is:

rsync -rvOlt --exclude '.DS_Store'                                  \
      --exclude '.com.apple.timemachine.supported'                  \
      --delete /Volumes/Storage/Music/Albums/ 192.168.1.100::albums

I have tried enabling -vvv and --progress, but there is no pattern that I can discern between what hangs and what doesn't. Heck, if I retry, the same file might hang at a different point during the transfer or not at all. A dry run (-n) does not always succeed either. The only "success" I've had is implementing a timeout (--timeout=10) and rerunning the command over and over. Eventually, I creep along, but with no guarantee of success and at a pace that is unacceptable. I've reached a point where I have one file that I can't get past.

The Mac Mini is connected to my router via 5 GHz. The FreeNAS box is wired into that same router on a 100 mbit port. When transfers are actually going, rsync --progress reports 2.5-4 MB/s. According to --progress, a hang is literally just that—no data transfer is occurring as far as I can tell.

I need help with both the diagnostics and the solution.

Answer 1

I am using openSUSE 13.2 Linux, rsync version 3.1.1-2.4.1.x86_64, and I experienced similar problems, doing an rsync between my laptop and an external hard disk, with the destination device definitively having enough free space.

I thought I got an improvement omitting option -v, but after 10 minutes it was hanging again: strace said: select(5, [], [4], [], {60, 0}) = 0 (Timeout)

And with "iotop" I counld see confirm that the rsync processes did no significant disk IO any more.

Neither removing the -v option nor limiting the bandwidth using --bwlimit fixed the problem.

Answer 2

Try using --whole-file/-W. This command disables the rsync delta-transfer algorithm. That is what worked for us (WSL to OSX)

our full sync flags were -avWPle

(e was because we were using ssh, and that has to be the last flag)

Answer 3

Most likely not "your" problem, but I stumbled upon this question when I was researching a similar behavior:

I'm observing "hanging" when the target site has too much io load. e.G. on one of my small business servers, when someone is resyncing his IMAP account and downloading large batchs of data and a backup job runs that writes his data.

In this situation I notice a steep drop in performance for rsync. Noticeable in a high load value in top on the target machine, even though CPU and Mem are fine.

Waiting for the process to finish has helped every time or interrupting and attempting the rsync at a later time again.

Answer 4

I've been running into the same thing again and again and it seems to help if you drop the -v option (which is annoying if you need that output).

Answer 5

Holger Ohmacht aka h8ohmh / 8ohmh:

The problem lies in the filesystem buffer / usage of the interworking of harddisk/hw so far as I could investigate.

Temporal solution for local drives (eg. USB3<->HD) : A script which is polling the changing disk space. If no changing free disk space then rsync is stalled and has to be restarted

cmd="rsync -aW --progress --stats --preallocate --super \ <here your source dir> \ <here your dest dir>" eval "$cmd" &
rm ./ndf.txt rm ./odf.txt while [[ 0 == 0 ]]; do df > ./ndf.txt cmp ./odf.txt ./ndf.txt res="$?" echo "$res" if [[ $res == 0 ]]; then echo "###########################################" ls -al "./ndf.txt" ls -al "./odf.txt" killall rsync eval "$cmd" & else cp ./ndf.txt ./odf.txt fi sleep 60 done Change <source dir> etc to your paths!

In my case it is always stalling by usage of rsync's --preallocate option (normally because of better disk performance and rescueing continuous blocks), so as long as the disk and filesystem drivers not reworked there just this solution

Answer 6

In my situation rsync was not actually failing.

I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.

What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.

This is likely a non issue if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers.

On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, sollution is to add ClientAliveInterval 120 and ClientAliveCountMax 720 to /etc/ssh/sshd_config.