scp: how to find out that copying was finished

问题:

I'm using scp command to copy file from one Linux host to another. I run scp commend on host1 and copy file from host1 to host2. File is quite big and it takes for some time to copy it. On host2 file appears immediately as soon as copying was started. I can do everything with this file even if copying is still in progress.

Is there any reliable way to find out if copying was finished or not on host2?

回答1:

Off the top of my head, you could do something like:

touch tinyfile
scp bigfile tinyfile user@host:

Then when tinyfile appears you know that the transfer of bigfile is complete.

As pointed out in the comments, this assumes that scp will copy the files one by one, in the order specified. If you don't trust it, you could do them one by one explicitly:

scp bigfile user@host:
scp tinyfile user@host:

The disadvantage of this approach is that you would potentially have to authenticate twice. If this were an issue you could use something like ssh-agent.

回答2:

On sending side (host1) use script like this:

#!/bin/bash

echo 'starting transfer'
scp FILE USER@DST_SERVER:DST_PATH
OUT=$?
if [ $OUT = 0 ]; then
  echo 'transfer successful'
  touch successful
  scp successful USER@DST_SERVER:DST_PATH
else
  echo 'transfer faild'
fi

On receiving side (host2) make script like this:

#!/bin/bash 

SLEEP_TIME=30
MAX_CNT=10
CNT=0
while [[ ! -e successful && $CNT < $MAX_CNT ]]; do
    ((CNT++))

    sleep($SLEEP_TIME);
done; 

if [[ -e successful ]]; then
    echo 'successful'
    rm successful

    # do somethning with FILE
fi

With CNT and MAX_CNT you disable endless loop (in case file successful isn't transferred). Product MAX_CNT and SLEEP_TIME should be equal or greater expected transfer time. In my example expected transfer time is less than 300 seconds.

回答3:

A checksum (md5sum, sha256sum ,sha512sum) of the local and remote files would tell you if they're identical.

For the situation where you don't have SSH access to the remote system - like an FTP server - you can download the file after it's uploaded and compare the checksums. I do this for files I send from production scripts at work. Below is a snippet from the script in which I do this.

MD5SRC=$(md5sum $LOCALFILE | cut -c 1-32)
MD5TESTFILE=$(mktemp -p /ramdisk)
curl \
    -o $MD5TESTFILE \
    -sS \
    -u $FTPUSER:$FTPPASS \
    ftp://$FTPHOST/$REMOTEFILE
MD5DST=$(md5sum $MD5TESTFILE | cut -c 1-32)
if [ "$MD5SRC" == "$MD5DST" ]
then
    echo "+Local and Remote files match!"
else 
    echo "-Local and Remote files don't match"
fi

回答4:

After some investigation, and discussion of the problem on other forums I have found one more solution. Maybe it can help somebody.

There is a command "lsof". It lists open files. During copying the file will be opened, so the command

lsof | grep filename

will return non empty result.

So you might want to make a while loop to wait until lsof returns nothing and proceed with your task.

Example:

# provide your file name here
f=<nameOfYourFile>
lsofresult=`lsof | grep $f | wc -l`
while [ $lsofresult != 0 ]; do 
  echo still copying file $f...
  sleep 5
  lsofresult=`lsof | grep $f | wc -l`
done; echo copying file $f is finished: `ls $f`

回答5:

For the duplicate question, How to check if file has been scp 100% to the remote location , which was for an expect script, to know if a file is transferred completely, we can add expect 100% .. .. i.e something like this ...

expect -c "  
   set timeout 1
   spawn scp user@$REMOTE_IP:/tmp/my.file user@$HOST_IP:/home/.
   expect yes/no { send yes\r ; exp_continue }
   expect password: { send $SCP_PASSWORD\r }
   expect 100%
   sleep 1
   exit
"  
if [ -f "/home/my.file" ]; then  
echo "Success"  
fi

回答6:

If avoiding a second SSH handshake is important, you can use something like the following:

ssh host cat \> bigfile \&\& touch complete < bigfile

Then wait for the "complete" file to get created on the remote end.

回答7:

if you use inotify-tools, then the solution will looks like this:

while ! inotifywait -e close $(dirname ${bigfile_fullname}) 2>/dev/null | \
    grep -Eo "CLOSE $(basename ${bigfile_fullname})$">/dev/null
do true
done
echo "File ${bigfile_fullname} closed"