Compressing huge files (> 2GB) into ZIP on the cli

问题:

Im building an upload tool using node.js and socket.io, because they usually upload incredibly huge files and normal upload forms wouldnt work. The problem is that they wanted to compress the files into a zip before sending them, to increase efficiency in transmission.

I've been researching around compression methods like JSZip or zip.js, but neither of those work well with extremely big files. What could I do?

回答1:

You can compress up to 4GB of data with zip.js but:

• it will work only with Chrome
• it will be quite slow (approx. 30min per GB of compressed data on my old laptop)

You can try it online with this demo. You must select the "HDD" option in the "choose temporary storage" input. Then you can import big files from your filesystem and monitor the memory consumption: it should be stable (approx. 300MB on my laptop).

Selecting "HDD" means that zip.js will use File API: Directories and System to store compressed data. This API is currently only available on Chrome and allows to write data into a sandboxed virtual filesystem. The demo uses the temporary storage which does not require a user permission.

Edit: You could also implement your own Writer constructor function to stream the data to your server while zip.js compresses it: it wouldn't rely on the filesystem API and should work on every browser you support. A Writer must just implement these 2 methods:

init(callback[, onerror])
// initializes the stream where to write the zipped data

writeUint8Array(array, callback[, onerror])
// writes asynchronously a Uint8Array into the stream.

// getData method is optional

Here is an example of custom Writer and Reader constructors. You can also look at zip.js Writers implementations for more examples.

回答2:

If you're looking for a compressor on the client side I'm sorry to tell you that JS is not the way to go.

A side from that, your best bet is to tell your users to just compress the files before uploading them. Or even use some other kind of tool (FTP maybe?).

EDIT:

Oh, btw, ZIP is really inefficient with random data, so you'll be wasting CPU time for the compression/decompression and you won't shrink almost nothing in size.