OutOfMemoryException when creating large ZIP file

Question

I am trying to debug an OutOfMemoryException that occurs when creating a fairly large ZIP file using System.IO.Packaging.ZipPackage.

The code is iterating through a large list of objects, doing the following for each object.

1. Serializing the object data to a temporary file.
2. Creating a PackagePart for the file.
3. Copy from a source System.IO.Stream to another:
   • Source stream: FileStream
   • Target stream: PackagePart::GetStream() => MS.Internal.IO.Zip.ZipIOModeEnforcingStream

Finally it calls Package::Close() which saves the file.

The problem I am having is that for a particularly large list of objects, I am seeing an OutOfMemoryException (the x86 process size is getting to about 1.2GB in size).

I was thinking about partitioning the object data into chunks so I only process a smaller amount per loop (i.e. steps 1-3 above). The idea is that I would create n ZIP files in a temporary directory, and then find a way to combine them into a single archive.

Is this possible using System.IO.Packaging? What would I use to combine the parts?

Or is there a better way to fix this?

Answer 1

Calling the Flush method on the Package object in between creating a new package should probably solve the problem as that would cause the memory buffer to be flushed to disk.

Answer 2

I would use the DotNetZip library (http://dotnetzip.codeplex.com/). I've tried several zip libraries (System.IO like you are currently using and also SharpZibLib) and by far the easiest to use is the DotNetZip library.

You'll almost certainly end up with fewer lines of code and I found the memory usage to be very good (had a problem in a virtual machine environment which I reported and a new release fixed it).