Memory usage when creating and downloading zip arc

Question

I have a web api GET method that returns a zip file for downloading. Here's the code that creates the zip archive:

var resultStream = new MemoryStream();    
using (var zipArchive = new ZipArchive(resultStream, ZipArchiveMode.Create, leaveOpen: true))
{
    foreach (var file in files)
    {
        zipArchive.CreateEntryFromFile(file.Path, file.Name, CompressionLevel.Optimal);
    }
}

And here's how the response gets populated:

var response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new ByteArrayContent(resultStream.ToArray());
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zip");
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = "export_" + DateTime.Now.ToString("dd-MM-yyyy_HH-mm-ss") + ".zip";
response.Content.Headers.ContentDisposition.CreationDate = DateTime.Now;
response.Content.Headers.ContentDisposition.Size = resultStream.Length;
response.Content.Headers.ContentLength = resultStream.Length;

The code above works just fine, the problem is it consumes a lot of memory on the server, depending of course on the file sizes. I've tried changing the result to StreamContent, however this didn't work as the response only returned headers and eventually timed out.

So here are my questions:

1. Is there a way to avoid loading all files in memory and instead send the zip file as it gets created?
2. Is using StreamContent better to use in this scenario and if yes, what do I need to change to make it work?
3. How is buffering affecting memory consumption in each case? I've tried disabling buffering by implementing a custom IHostBufferPolicySelector as suggested in this article, but it doesn't appear to have any effect.
4. The api action currently can be called by navigating a link, using HttpClient or by AJAX request, so any solution has to support all scenarios.

Answer 1

Adapted from the Kudu project, a method that uses PushStreamContent in combination with a specific DelegatingStream wrapper to stream a zip archive:

public static class ZipStreamContent
{
    public static PushStreamContent Create(string fileName, Action<ZipArchive> onZip)
    {
        var content = new PushStreamContent((outputStream, httpContent, transportContext) =>
        {
            using (var zip = new ZipArchive(new StreamWrapper(outputStream), ZipArchiveMode.Create, leaveOpen: false))
            {
                onZip(zip);
            }
        });
        content.Headers.ContentType = new MediaTypeHeaderValue("application/zip");
        content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
        content.Headers.ContentDisposition.FileName = fileName;
        return content;        
    }

    // this wraps the read-only HttpResponseStream to support ZipArchive Position getter.
    public class StreamWrapper : DelegatingStream
    {
        private long _position = 0;

        public StreamWrapper(Stream stream)
            : base(stream)
        {
        }

        public override long Position
        {
            get { return _position; }
            set { throw new NotSupportedException(); }
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            _position += count;
            base.Write(buffer, offset, count);
        }

        public override IAsyncResult BeginWrite(byte[] buffer, int offset, int count, AsyncCallback callback, object state)
        {
            _position += count;
            return base.BeginWrite(buffer, offset, count, callback, state);
        }
    }
}

Which for your case you could use like:

var response = new HttpResponseMessage(HttpStatusCode.OK);
var response.Content = ZipStreamContent.Create(
    "export_" + DateTime.Now.ToString("dd-MM-yyyy_HH-mm-ss") + ".zip",
    zipArchive => {
        foreach (var file in files)
        {
            zipArchive.CreateEntryFromFile(file.Path, file.Name, CompressionLevel.Optimal);
        }        
    });