broken pipe with remote rsync between two servers

Question

I am trying to transfer a large dataset (768 Gigs) from one remote machine to another using bash on ubuntu 16.04. The problem I appear to be having is that I use rsync and the machine will transfer for a few hours and then quit when the connection inevitably gets interrupted. So suppose Im on machine A and the remote servers are machines B and C (all machines using ubuntu 16.04). I ssh to machine B and use this command:

nohup rsync -P -r -e ssh /path/to/files/on/machine_B user@machine_C:directory &

note that I have the authorized key setup so no password is required between machines B and C

A few hours later I get the following in the nohup file:

sending incremental filelist
file_1.bam
90,310,583,648 100%   36.44MB/s    0:39:23 (xfr#4, to-chk=5/10)
file_2.bam
79,976,321,885 100%   93.25MB/s    0:13:37 (xfr#3, to-chk=6/10)
file_3.bam
88,958,959,616  88%   12.50MB/s    0:15:28  rsync error: unexplained error (code 129) at rsync.c(632) [sender=3.1.1]
rsync: [sender] write error: Broken pipe (32)

I used nohup because I though it would keep running even if there was a hangup. I have not tried sh -c and I have not tried running the command from machine A because at this point whatever I try would be guesswork, ideas would be appreciated.

Answer 1

for those that are interested I also tried running the following script with the nohup command on machine B.

script:

chomp( my @files = `ls /path/to/files/on/machineB/*` );
foreach ( @files ) { system("scp $_ user@machineC:destination/"); }

I still got truncated files.

at the moment the following command appears to be working:

nohup rsync -P --append -r -e ssh /path/to/files/on/machine_B user@machine_C:directory &

you just have to check the nohup file for a broken pipe error and re-enter the command if necessary.