Good way to send a large file over a network in C#

Question

I am trying to build an application that can request files from a service running on another machine in the network. These files can be fairly large (500mb + at times). I was looking into sending it via TCP but I'm worried that it may require that the entire file be stored in memory.

There will probably only be one client. Copying to a shared directory isn't acceptable either. The only communication required is for the client to say "gimme xyz" and the server to send it (and whatever it takes to ensure this happens correctly).

Any suggestions?

Answer 1

Use TransmitFile (which is a Win32 function; perhaps it's a method of the .NET library as well).

Answer 2

Personally I'd go for something that balances speed, reliability and economical code, so I'd base it on a TCP network stream. The client-side of the code would look like this:

internal class Client
{
    private FileStream _fs;     
    private long _expectedLength;

    public void GetFileFromServer(string localFilename)
    {            
        if (File.Exists(localFilename))
            File.Delete(localFilename);

        _fs = new FileStream(localFilename, FileMode.Append);

        var ipEndpointServer = new IPEndPoint(IPAddress.Parse({serverIp}), {serverPort});

        // an object that wraps tcp client
        var client = new TcpClientWrapper(ipEndpointServer, "");
        client.DataReceived += DataReceived;
    }

    private void DataReceived(object sender, DataReceivedEventArgs e)
    {
        var data = e.Data;

        // first packet starts with 4 bytes dedicated to the length of the file
        if (_expectedLength == 0)
        {
            var headerBytes = new byte[4];
            Array.Copy(e.Data, 0, headerBytes, 0, 4);
            _expectedLength = BitConverter.ToInt32(headerBytes, 0);
            data = new byte[e.Data.Length - 4];
            Array.Copy(e.Data, 4, data, 0, data.Length);
        }

        _fs.WriteAsync(e.Data, 0, e.Data.Length);

        if (_fs.Length >= _expectedLength)
        {                                
            // transfer has finished
        }
    }
}

Then have a server class to serve the file. Note how the whole file isn't loaded into memory, instead it is read in chunks from a FileStream.

internal class Server
{
    private TcpServer _tcpServer;
    private NetworkStream _stream;        

    public void StartServer()
    {
        // fire up a simple Tcp server
        _tcpServer = new TcpServer({serverPort}, "test");
        _tcpServer.ClientConnected += ClientConnected;
    }

    private void ClientConnected(object sender, TcpClientConnectedEventArgs e)
    {            
        // an incoming client has been detected ... send the file to that client!
        _stream = e.Client.GetStream();
        SendFileToClient({pathToFile});
    }

    private void SendFileToClient(string pathToFile)
    {
        // open the file as a stream and send in chunks
        using (var fs = new FileStream(pathToFile, FileMode.Open))
        {
            // send header which is file length
            var headerBytes = new byte[4];
            Buffer.BlockCopy(BitConverter.GetBytes(fs.Length + 4), 0, headerBytes, 0, 4);
            _stream.Write(headerBytes, 0, 4);

            // send file in block sizes of your choosing
            var buffer = new byte[100000];
            int bytesRead = 0;
            while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)
            {
                _stream.Write(buffer, 0, bytesRead);
            }
            _stream.Flush();
        }
    }
}

The TcpClientWrapper is pretty much boiler plate code with the System.Net.Sockets.TcpClient object and the underlying NetworkStream object. I don't really need to post this as well, but just to give some pointers ther construction would contain something like this:

_tcp = new Net.TcpClient();
_tcp.Connect(remoteEp);
_stream = _tcp.GetStream();
_stream.BeginRead(_receivedData, 0, _receivedData.Length, DataReceivedAsync, null);

and the DataReceivedAsync method is boilerplate socket data handling and would raise an event o share the received data back to the consumer (the client in this case):

private void DataReceivedAsync(IAsyncResult ar)
{
    var receivedBytes = _stream.EndRead(ar);

    if (receivedBytes > 0)
    {
        var data = new byte[receivedBytes];
        Array.Copy(_receivedData, 0, data, 0, receivedBytes);
        DataReceived?.Invoke(this, new DataReceivedEventArgs(data));

        _receivedData = new byte[ReceiveBufferSize];
        _stream.BeginRead(_receivedData, 0, _receivedData.Length, DataReceivedAsync, null);
    }       
}   

The event to ship data from the wrapper back to the client:

public EventHandler<DataReceivedEventArgs> DataReceived;
public class DataReceivedEventArgs : EventArgs
{
    public DataReceivedEventArgs(byte[] data) { Data = data; }
    public byte[] Data { get; }
}

Answer 3

Be careful with BITS. It is a very good protocol but not a critical part of the windows update program. We found that hardly any of our corporate clients allowed the BITS update onto their machines; therefore we couldn't build an app that relied on it.

Answer 4

Use FTP via the open source edtFTPnet library. Fast and simple.

Answer 5

If FTP was an option then I'd go with that for the sake of simplicity. Otherwise you're into a world of TCP/IP socket programming.

Answer 6

Here is an easier way. Using BITS (Background Intelligent Transfer Service). Its already built into WinXP and Vista. Its basically what drives Windows Updates.

http://blogs.msdn.com/powershell/archive/2009/01/11/transferring-large-files-using-bits.aspx

http://blogs.msdn.com/jamesfi/archive/2006/12/23/how-to-use-bits-to-transfer-files.aspx

Here is a nice managed BITS wrapper someone wrote and how to use it.

http://www.codeproject.com/KB/cs/Managed_BITS.aspx