Memory buffers fills up when I stream upload a fil

Question

Trying to stream-upload to S3 using skipper and skipper-s3 using this code

image_upload : function  (req,res) {

    // depends on bodyparser:skipper in confing/http.js

    console.log("Uploading image .. ");

    req.file('image')
        .upload({
            adapter: require('skipper-s3'),
            key: KEY,
            secret: SECRET,
            bucket: 'mybucket',
            tmpdir: 'tmp/s3-multiparts'
        }, function whenDone(err, uploadedFiles) {
            if (err){
                return res.negotiate(err);
            } else {
                dict = {
                    files: uploadedFiles,
                    textParams: req.params.all()
                }
                return res.ok(dict);
            }
        }

    );
};

I have 2 problems:

First: when I try to upload some file, I can see that this file buffers in memory, instead just buffering to the disk (path added to options).

Actually this is a big problem, as my memory will fill up so quickly when having many users concurrently uploading

Second: my application crashes when upload is bigger than 5MBs, giving me this error

events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Request aborted
    at IncomingMessage.onReqAborted (~Desktop/myApp/node_modules/sails/node_modules/skipper/node_modules/multiparty/index.js:175:17)
    at IncomingMessage.emit (events.js:104:17)
    at abortIncoming (_http_server.js:279:11)
    at Socket.serverSocketCloseListener (_http_server.js:292:5)
    at Socket.emit (events.js:129:20)
    at TCP.close (net.js:485:12)

Answer 1

Solution to your second problem: You can specify maximum size of file to be uploaded. Default is 5 MB.

req.file('some_file').upload({
    dirname: 'path to store the file',/* optional. defaults to assets/uploads I guess*/
    saveAs: 'new file name', /* optional. default file name */
    maxBytes: 5 * 1024 * 1024 //5 MB
}, function(err, uploadedFiles) {

});

I am not sure about the first problem though. You don't need to specify a tempdir in options. Skipper-S3 by default writes the file to <your project root>/.tmp/s3-upload-part-queue for its upload purpose. I don't see why the memory should get filled up. Can you possibly remove the tempdir and check?