Download new and modified files from an FTP server

Question

I'm trying to get a list of the files on an FTP server, then one by one check if that file exists on the local system and if it does compare the dates modified and if the ftp file is newer download it.

private void btnGo_Click(object sender, EventArgs e)
{
    string[] files = GetFileList();
    foreach (string file in files)
    {
        if (file.Length >= 5)
        {
            string uri = "ftp://" + ftpServerIP + "/" + remoteDirectory + "/" + file;
            Uri serverUri = new Uri(uri);

            CheckFile(file);
        }
    }
    this.Close();
}

public string[] GetFileList()
{
    string[] downloadFiles;
    StringBuilder result = new StringBuilder();
    WebResponse response = null;
    StreamReader reader = null;

    try
    {
        FtpWebRequest reqFTP = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://" + ftpServerIP + "/" + remoteDirectory));
        reqFTP.UseBinary = true;
        reqFTP.Credentials = new NetworkCredential(ftpUserID, ftpPassword);
        reqFTP.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
        reqFTP.Proxy = null;
        reqFTP.KeepAlive = false;
        reqFTP.UsePassive = false;
        response = reqFTP.GetResponse();
        reader = new StreamReader(response.GetResponseStream());

        string line = reader.ReadLine();
        while (line != null)
        {
            result.Append(line);
            result.Append("\n");
            line = reader.ReadLine();
        }
        result.Remove(result.ToString().LastIndexOf('\n'), 1);
        return result.ToString().Split('\n');
    }
    catch
    {
        if (reader != null)
        {
            reader.Close();
        }
        if (response != null)
        {
            response.Close();
        }
        downloadFiles = null;
        return downloadFiles;
    }
}

private void CheckFile(string file)
{
    string dFile = file;
    string[] splitDownloadFile = Regex.Split(dFile, " ");
    string fSize = splitDownloadFile[13];
    string fMonth = splitDownloadFile[14];
    string fDate = splitDownloadFile[15];
    string fTime = splitDownloadFile[16];
    string fName = splitDownloadFile[17];


    string dateModified = fDate + "/" + fMonth+ "/" + fYear;

    DateTime lastModifiedDF = Convert.ToDateTime(dateModified);

    string[] filePaths = Directory.GetFiles(localDirectory);

    // if there is a file in filePaths that is the same as on the server compare them and then download if file on server is newer
    foreach (string ff in filePaths)
    {

        string[] splitFile = Regex.Split(ff, @"\\");

        string fileName = splitFile[2];
        FileInfo fouFile = new FileInfo(ff);
        DateTime lastChangedFF = fouFile.LastAccessTime;
        if (lastModifiedDF > lastChangedFF) Download(fileName);
    }
}

In the check file method, for each file (they are .exe files) I keep getting different results when I split the string i.e. for one file the file name was at column 18 another it was at 16 etc. I also can't always get the year portion of the file.

Answer 1

Option A: I'd recommend that you use a higher-level FTP client library that handles some of these details for you, a few likely options are:

• http://ftplib.codeplex.com/
• http://ftps.codeplex.com/
• http://netftp.codeplex.com/

Option B: To answer your question more directly, I think the issue is with this line:

string[] splitDownloadFile = Regex.Split(dFile, " ");

It seems like the FTP server is using spaces to right-align the filenames. To deal with that, we want to adjust the regex to consume all whitespace between the fields:

string[] splitDownloadFile = Regex.Split(dFile, "\s+");

...where \s stands for any whitespace character (usually tabs or spaces), and + means one or more of the thing to the left of it. This will not handle edge cases, such as file names with spaces in them.

Answer 2

First of all, there are some components for which you can get info and download data from ftp can be found here: http://www.limilabs.com/ftp

I wrote some methods for get filename and last modified date from ftp.

This is how I get the filename from line:

private string GetFtpName(string line)
{
    for (int i = 0; i < 8; i++)
        line = line.Substring(line.IndexOf(" ")).Trim();
    return line;
}

And this is how I get the last modified date from ftp:

private DateTime GetFtpFileDate(string url, ICredentials credential)
{
    FtpWebRequest rd = (FtpWebRequest)WebRequest.Create(url);
    rd.Method = WebRequestMethods.Ftp.GetDateTimestamp;
    rd.Credentials = credential;

    FtpWebResponse response = (FtpWebResponse)rd.GetResponse();
    DateTime lmd = response.LastModified;
    response.Close();

    return lmd;
}

Answer 3

Here is an exerpt from the FTPclient source which shows you how they are building their. FtpFileInfo objects. I'm not able to test to this make sure this will work in all cases at the moment, but maybe it will give you some ideas.

    /// <summary>
    /// Return a detailed directory listing, and also download datetime stamps if specified
    /// </summary>
    /// <param name="directory">Directory to list, e.g. /pub/etc</param>
    /// <param name="doDateTimeStamp">Boolean: set to True to download the datetime stamp for files</param>
    /// <returns>An FTPDirectory object</returns>
    public FTPdirectory ListDirectoryDetail(string directory, bool doDateTimeStamp)
    {
        System.Net.FtpWebRequest ftp = GetRequest(GetDirectory(directory));
        // Set request to do simple list
        ftp.Method = System.Net.WebRequestMethods.Ftp.ListDirectoryDetails;

        string str = GetStringResponse(ftp);
        // replace CRLF to CR, remove last instance
        str = str.Replace("\r\n", "\r").TrimEnd('\r');
        // split the string into a list
        FTPdirectory dir = new FTPdirectory(str, _lastDirectory);

        // download timestamps if requested
        if (doDateTimeStamp)
        {
            foreach (FTPfileInfo fi in dir)
            {
                fi.FileDateTime = this.GetDateTimestamp(fi);
            }
        }

        return dir;
    }

    /// <summary>
    /// Obtain datetimestamp for remote file
    /// </summary>
    /// <param name="filename"></param>
    /// <returns></returns>
    public DateTime GetDateTimestamp(string filename)
    {
        string path;
        if (filename.Contains("/"))
        {
            path = AdjustDir(filename);
        }
        else
        {
            path = this.CurrentDirectory + filename;
        }
        string URI = this.Hostname + path;
        FtpWebRequest ftp = GetRequest(URI);
        ftp.Method = WebRequestMethods.Ftp.GetDateTimestamp;
        return this.GetLastModified(ftp);
    }

Answer 4

For this, you need to retrieve a remote directory listing, including timestamps.

Unfortunately, there's no really reliable and efficient way to retrieve timestamps using features offered by the .NET framework as it does not support the FTP MLSD command. The MLSD command provides listing of remote directory in a standardized machine-readable format. The command and the format is standardized by the RFC 3659.

Alternatives you can use, that are supported by the .NET framework (as the other answers show):

• the ListDirectoryDetails method (the FTP LIST command) to retrieve details of all files in a directory and then you deal with FTP server specific format of the details (*nix format similar to ls *nix command is the most common, drawback is that the format may change over time, as for newer files "May 8 17:48" format is used and for older files "Oct 18 2009" format is used)

  DOS/Windows format: C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response
  *nix format: Parsing FtpWebRequest ListDirectoryDetails line

• the GetDateTimestamp method (an FTP MDTM command) to individually retrieve timestamps for each file. An advantage is that the response is standardized by the RFC 3659 to YYYYMMDDHHMMSS[.sss]. A disadvantage is that you have to send a separate request for each file, what can be quite inefficient.

  const string uri = "ftp://ftp.example.com/remote/path/file.txt";
  FtpWebRequest request = (FtpWebRequest)WebRequest.Create(uri);
  request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
  FtpWebResponse response = (FtpWebResponse)request.GetResponse();
  Console.WriteLine("{0} {1}", uri, response.LastModified);
  

---

Alternatively you can use a 3rd party FTP client implementation that supports the modern MLSD command.

For example the WinSCP .NET assembly supports that.

You can use the Session.ListDirectory or the Session.EnumerateRemoteFiles methods and read the RemoteFileInfo.LastWriteTime of the files in returned collection.

Or even easier, you can use the Session.SynchronizeDirectories to have the library automatically download (synchronize) the modified files:

// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
    Protocol = Protocol.Ftp,
    HostName = "ftp.example.com",
    UserName = "user",
    Password = "mypassword",
};

using (Session session = new Session())
{
    // Connect
    session.Open(sessionOptions);

    // Synchronize files
    session.SynchronizeDirectories(
        SynchronizationMode.Local, @"d:\www", "/remote/path", false).Check();
}

(I'm the author of WinSCP)

Answer 5

Try

ListDirectory + GetDateTimestamp 

instead of

ListDirectoryDetails