Downloading all files using FTP and C#

Question

What is the best way to download all files in a remote directory using C# and FTP and save them to a local directory?

Thanks.

Answer 1

Using C# FtpWebRequest and FtpWebReponse, you can use the following recursion (make sure the folder strings terminate in '\'):

    public void GetAllDirectoriesAndFiles(string getFolder, string putFolder)
    {
        List<string> dirIitems = DirectoryListing(getFolder);
        foreach (var item in dirIitems)
        {
            if ( item.Contains('.')  )
            {
                GetFile(getFolder + item, putFolder + item);
            }
            else
            {
                var subDirPut = new DirectoryInfo(putFolder + "\\" + item);
                subDirPut.Create();
                GetAllDirectoriesAndFiles(getFolder + item + "\\", subDirPut.FullName + "\\");
            }
        }
    }

The "item.Contains('.')" is a bit primitive, but has worked for my purposes. Post a comment if you need an example of the methods:

GetFile(string getFileAndPath, string putFileAndPath)

or

DirectoryListing(getFolder)

Answer 2

For FTP protocol you can use FtpWebRequest class from .NET framework. Though it does not have any explicit support for recursive file operations (including downloads). You have to implement the recursion yourself:

• List the remote directory
• Iterate the entries, downloading files and recursing into subdirectories (listing them again, etc.)

Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the FtpWebRequest. The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.

Your options are:

• Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it's a directory. But that can become a performance problem, when you have a large number of entries.
• You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
• You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)

void DownloadFtpDirectory(string url, NetworkCredential credentials, string localPath)
{
    FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url);
    listRequest.UsePassive = true;
    listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
    listRequest.Credentials = credentials;

    List<string> lines = new List<string>();

    using (WebResponse listResponse = listRequest.GetResponse())
    using (Stream listStream = listResponse.GetResponseStream())
    using (StreamReader listReader = new StreamReader(listStream))
    {
        while (!listReader.EndOfStream)
        {
            lines.Add(listReader.ReadLine());
        }
    }

    foreach (string line in lines)
    {
        string[] tokens =
            line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
        string name = tokens[8];
        string permissions = tokens[0];

        string localFilePath = Path.Combine(localPath, name);
        string fileUrl = url + name;

        if (permissions[0] == 'd')
        {
            Directory.CreateDirectory(localFilePath);
            DownloadFtpDirectory(fileUrl + "/", credentials, localFilePath);
        }
        else
        {
            FtpWebRequest downloadRequest = (FtpWebRequest)WebRequest.Create(fileUrl);
            downloadRequest.UsePassive = true;
            downloadRequest.UseBinary = true;
            downloadRequest.Method = WebRequestMethods.Ftp.DownloadFile;
            downloadRequest.Credentials = credentials;

            using (Stream ftpStream = downloadRequest.GetResponse().GetResponseStream())
            using (Stream fileStream = File.Create(localFilePath))
            {
                ftpStream.CopyTo(fileStream);
            }
        }
    }
}

The url must be like:

• ftp://example.com/ or
• ftp://example.com/path/

---

Or use 3rd party library that supports recursive downloads.

For example with WinSCP .NET assembly you can download whole directory with a single call to Session.GetFiles:

// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
    Protocol = Protocol.Ftp,
    HostName = "example.com",
    UserName = "user",
    Password = "mypassword",
};

using (Session session = new Session())
{
    // Connect
    session.Open(sessionOptions);

    // Download files
    session.GetFiles("/home/user/*", @"d:\download\").Check();
} 

Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.

(I'm the author of WinSCP)

Answer 3

You can use FTPClient from laedit.net. It's under Apache license and easy to use.

It use FtpWebRequest :

• first you need to use WebRequestMethods.Ftp.ListDirectoryDetails to get the detail of all the list of the folder
• for each files you need to use WebRequestMethods.Ftp.DownloadFile to download it to a local folder

Answer 4

You could use System.Net.WebClient.DownloadFile(), which supports FTP. MSDN Details here

Answer 5

downloading all files in a specific folder seems to be an easy task. However, there are some issues which has to be solved. To name a few:

• How to get list of files (System.Net.FtpWebRequest gives you unparsed list and directory list format is not standardized in any RFC)
• What if remote directory has both files and subdirectories. Do we have to dive into the subdirs and download it's content?
• What if some of the remote files already exist on the local computer? Should they be overwritten? Skipped? Should we overwrite older files only?
• What if the local file is not writable? Should the whole transfer fail? Should we skip the file and continue to the next?
• How to handle files on a remote disk which are unreadable because we don’t have sufficient access rights?
• How are the symlinks, hard links and junction points handled? Links can easily be used to create an infinite recursive directory tree structure. Consider folder A with subfolder B which in fact is not the real folder but the *nix hard link pointing back to folder A. The naive approach will end in an application which never ends (at least if nobody manage to pull the plug).

Decent third party FTP component should have a method for handling those issues. Following code uses our Rebex FTP for .NET.

using (Ftp client = new Ftp())
        {
            // connect and login to the FTP site
            client.Connect("mirror.aarnet.edu.au");
            client.Login("anonymous", "my@password");

            // download all files
            client.GetFiles(
                "/pub/fedora/linux/development/i386/os/EFI/*",
                "c:\\temp\\download",
                FtpBatchTransferOptions.Recursive,
                FtpActionOnExistingFiles.OverwriteAll
            );

            client.Disconnect();
        }

The code is taken from my blogpost available at blog.rebex.net. The blogpost also references a sample which shows how ask the user how to handle each problem (e.g. Overwrite/Overwrite older/Skip/Skip all).