How to quickly zip large files in PHP

Question

I wrote a PHP script to dynamically pack files selected by the client into zip file and force a download. It works well except that when the number of files is huge (like over 50000), it takes a very long time for the download dialog box to appear on the client side.

I thought about improving this using cache (these files are not changed very often), but because the selection of the files are totally decided by the user, and there are tens of thousands of combinations on the selection, it is very hard to cache combinations. I also thought about generating zip archives for individual files first, and then combining the zip files on-the-fly. But I did't find a way to concatenate zip files in PHP. Another way I can think of is sending (i.e., reading) the zip file at the same time as generating it. I also don't know if this is supported.

If someone could help me on this, I would really appreciate your help.

Answer 1

To extened Mike Sherov's answer, try using a combination of Tar and Gzip/Zip. Individually pre-compress all the files using Gzip/Zip, Then when the client makes their selection, you simply Tar those files together. That way you still get the benefit of compression and the simplicity of downloading one file, but none of the overheads and delays associated with compressing large files in real time.

Answer 2

Check out mod_zip for Nginx:

https://github.com/evanmiller/mod_zip

It streams a ZIP file to the client dynamically and can include very large (2GB+) files while using very little RAM.

Answer 3

While not a silver bullet, you can try tar'ing the files instead. The resulting file is larger, but compression time is much shorter. See here for more info: http://birdhouse.org/blog/2010/03/08/zip-vs-tar-gzip/