FTP: copy, check integrity and delete

Question

I am looking for a way to connect to a remote server with ftp or lftp and make sure the following steps:

1. Copy files from FTP server to my local machine.

2. Check if the downloaded files are fine (i.e. md5checksum).

3. If the download was fine then delete the downloaded files from the FTP server.

This routine will be executed each day from my local machine. What would be the best option to do this? Is there a tool that makes abstraction of all the 3 steps ?

I am running Linux on both client and server machines.

Update: Additionally, I have also a text file that contains the association between the files on the FTPserver and their MD5sum. They were computed at the FTP server side.

Answer 1

First, make sure your remote server supports the checksum calculation at all. Many do not. I believe there's even no standard FTP command to calculate a checksum of a remote file. There were many proposals and there are many proprietary solutions.

The latest proposal is:
https://tools.ietf.org/html/draft-bryan-ftpext-hash-02

So even if your server supports checksum calculation, you have to find a client that supports the same command.

Some of the commands that can be used to calculate checksum are: XSHA1, XSHA256, XSHA512, XMD5, MD5, XCRC and HASH.

You can test that with WinSCP. The WinSCP supports all the previously mentioned commands. Test its checksum calculation function or the checksum scripting command. If they work, enable logging and check, what command and what syntax WinSCP uses against your server.

---

Neither the ftp (neither Windows nor *nix version) nor the lftp support checksum calculation, let only automatic verification of downloaded file.

I'm not even aware of any other client that can automatically verify downloaded file.

You can definitely script it with a help of some feature-rich client.

---

I've wrote this answer before OP specified that he/she is on Linux. I'm keeping the Windows solution in case it helps someone else.

On Windows, you could script it with PowerShell using WinSCP .NET assembly.

param (
    $sessionUrl = "ftp://username:[e-mail]/",
    [Parameter(Mandatory)]
    $localPath,
    [Parameter(Mandatory)]
    $remotePath,
    [Switch]
    $pause = $False
)

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path (Join-Path $PSScriptRoot "WinSCPnet.dll")

    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions
    $sessionOptions.ParseUrl($sessionUrl);

    $session = New-Object WinSCP.Session

    try
    {
        # Connect
        $session.Open($sessionOptions)

        Write-Host "Downloading $remotePath to $localPath..."
        $session.GetFiles($remotePath, $localPath).Check();

        # Calculate remote file checksum
        $buf = $session.CalculateFileChecksum("sha-1", $remotePath)
        $remoteChecksum = [BitConverter]::ToString($buf)
        Write-Host "Remote file checksum:     $remoteChecksum"

        # Calculate local file checksum
        $sha1 = [System.Security.Cryptography.SHA1]::Create()
        $localStream = [System.IO.File]::OpenRead($localPath)
        $localChecksum = [BitConverter]::ToString($sha1.ComputeHash($localStream))
        Write-Host "Downloaded file checksum: $localChecksum"

        # Compare cheksums
        if ($localChecksum -eq $remoteChecksum)
        {
            Write-Host "Match, deleting remote file"

            $session.RemoveFiles($remotePath).Check();
            $result = 0
        }
        else
        {
            Write-Host "Does NOT match"
            $result = 1
        }
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
}
catch [Exception]
{
    Write-Host "Error: $($_.Exception.Message)"
    $result = 1
}

# Pause if -pause switch was used
if ($pause)
{
    Write-Host "Press any key to exit..."
    [System.Console]::ReadKey() | Out-Null
}

exit $result

You can run it like:

powershell -file checksum.ps1 -remotePath ./file.dat -localPath C:\path\file.dat

This is partially based on WinSCP example for Verifying checksum of a remote file against a local file over SFTP/FTP protocol.

(I'm the author on WinSCP)

Answer 2

That's a long shot, but if the server supports php, you can exploit that.

Save the following as a php file (say, check.php), in the same folder as your name_of_file.txt file:

<? php
echo md5_file('name_of_file.txt');
php>

Then, visit the page check.php, and you should get the md5 hash of your file.

Related questions:

• Calculate file checksum in FTP server using Apache FtpClient
• How to perform checksums during a SFTP file transfer for data integrity?
• https://serverfault.com/q/98597/401691