Download large file in small chunks in C#

问题:

I need to download some file which is more than 25 MB large, but my network only allow to request a file of 25 MB only.

I am using following code

  const long DefaultSize = 26214400;
    long Chunk = 26214400;
    long offset = 0;
    byte[] bytesInStream;
    public void Download(string url, string filename)
    {
        long size = Size(url);
        int blocksize = Convert.ToInt32(size / DefaultSize);
        int remainder = Convert.ToInt32(size % DefaultSize);
        if (remainder > 0) { blocksize++; }

        FileStream fileStream = File.Create(@"D:\Download TEST\" + filename);
        for (int i = 0; i < blocksize; i++)
        {
            if (i == blocksize - 1)
            {
                Chunk = remainder;

            }

            HttpWebRequest req = (HttpWebRequest)System.Net.WebRequest.Create(url);
            req.Method = "GET";
            req.AddRange(Convert.ToInt32(offset), Convert.ToInt32(Chunk+offset));
            HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
            // StreamReader sr = new StreamReader(resp.GetResponseStream());


            using (Stream responseStream = resp.GetResponseStream())
            {
                bytesInStream = new byte[Chunk];
                responseStream.Read(bytesInStream, 0, (int)bytesInStream.Length);
                // Use FileStream object to write to the specified file
                fileStream.Seek((int)offset, SeekOrigin.Begin);
                fileStream.Write(bytesInStream,0, bytesInStream.Length);
            }
            offset += Chunk;

        }
        fileStream.Close();

    }
    public long Size(string url)
    {
        System.Net.WebRequest req = System.Net.HttpWebRequest.Create(url);
        req.Method = "HEAD";
        System.Net.WebResponse resp = req.GetResponse();
        resp.Close();
        return resp.ContentLength;

    }

It is properly writing content on disk but content is not working

回答1:

You should check how much was read before write, something like this (and you don't need to remember the offset to seek, the seek is automatic when you write):

int read;
do
{
    read = responseStream.Read(bytesInStream, 0, (int)bytesInStream.Length);
    if (read > 0)
        fileStream.Write(bytesInStream, 0, read);
}
while(read > 0);

回答2:

There is a similar SO questions that might help you Segmented C# file downloader

and

How to open multiple connections to download single file?

Also this code project article http://www.codeproject.com/Tips/307548/Resume-Suppoert-Downloading

回答3:

Range is zero based and you should subtract 1 from upper bound.

request.Headers.Range = new System.Net.Http.Headers.RangeHeaderValue(offset, chunkSize + offset - 1);

I published correct code fragment at the following link: https://stackoverflow.com/a/48019611/1099716

回答4:

Akka streams can help download file in small chunks from a System.IO.Stream using multithreading. https://getakka.net/articles/intro/what-is-akka.html

The Download method will append the bytes to the file starting with long fileStart. If the file does not exist, fileStart value must be 0.

using Akka.Actor;
using Akka.IO;
using Akka.Streams;
using Akka.Streams.Dsl;
using Akka.Streams.IO;

private static Sink<ByteString, Task<IOResult>> FileSink(string filename)
{
    return Flow.Create<ByteString>()
        .ToMaterialized(FileIO.ToFile(new FileInfo(filename), FileMode.Append), Keep.Right);
}

private async Task Download(string path, Uri uri, long fileStart)
{
    using (var system = ActorSystem.Create("system"))
    using (var materializer = system.Materializer())
    {
       HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
       request.AddRange(fileStart);

       using (WebResponse response = request.GetResponse())
       {
           Stream stream = response.GetResponseStream();

           await StreamConverters.FromInputStream(() => stream, chunkSize: 1024)
               .RunWith(FileSink(path), materializer);
       }
    }
}