Large file upload with WebSocket

Question

I'm trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can't figure out how to write "send this slice of the file, release the resources used then repeat". I was hoping I could avoid using something like Flash/Silverlight for this.

Currently, I'm working with something along the lines of:

function FileSlicer(file) {
    // randomly picked 1MB slices,
    // I don't think this size is important for this experiment
    this.sliceSize = 1024*1024;  
    this.slices = Math.ceil(file.size / this.sliceSize);

    this.currentSlice = 0;

    this.getNextSlice = function() {
        var start = this.currentSlice * this.sliceSize;
        var end = Math.min((this.currentSlice+1) * this.sliceSize, file.size);
        ++this.currentSlice;

        return file.slice(start, end);
    }
}

Then, I would upload using:

function Uploader(url, file) {
    var fs = new FileSlicer(file);
    var socket = new WebSocket(url);

    socket.onopen = function() {
        for(var i = 0; i < fs.slices; ++i) {
            socket.send(fs.getNextSlice()); // see below
        }
    }
}

Basically this returns immediately, bufferedAmount is unchanged (0) and it keeps iterating and adding all the slices to the queue before attempting to send it; there's no socket.afterSend to allow me to queue it properly, which is where I'm stuck.

Answer 1

I believe the send() method is asynchronous which is why it will return immediately. To make it queue, you'd need the server to send a message back to the client after each slice is uploaded; the client can then decide whether it needs to send the next slice or a "upload complete" message back to the server.

This sort of thing would probably be easier using XMLHttpRequest(2); it has callback support built-in and is also more widely supported than the WebSocket API.

Answer 2

In order to serialize this operation you need the server to send you a signal every time a slice is received & written (or an error occurs), this way you could send the next slice in response to the onmessage event, pretty much like this:

function Uploader(url, file) {
    var fs = new FileSlicer(file);
    var socket = new WebSocket(url);

    socket.onopen = function() {
       socket.send(fs.getNextSlice());
    }
    socket.onmessage = function(ms){
        if(ms.data=="ok"){
           fs.slices--;
           if(fs.slices>0) socket.send(fs.getNextSlice());
        }else{
           // handle the error code here.
        }
    }
}

Answer 3

Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice().

This article helps you to handle large files in workers. change XHR send to Websocket in main thread.

//Messages from worker
function onmessage(blobOrFile) {
 ws.send(blobOrFile);
}

//construct file on server side based on blob or chunk information.

Answer 4

EDIT : The web world, browsers, firewalls, proxies, changed a lot since this answer was made. Right now, sending files using websockets can be done efficiently, especially on local area networks.

Websockets are very efficient for bidirectional communication, especially when you're interested in pushing information (preferably small) from the server. They act as bidirectional sockets (hence their name).

Websockets don't look like the right technology to use in this situation. Especially given that using them adds incompatibilities with some proxies, browsers (IE) or even firewalls.

On the other end, uploading a file is simply sending a POST request to a server with the file in the body. Browsers are very good at that and the overhead for a big file is really near nothing. Don't use websockets for that task.

Answer 5

You could use https://github.com/binaryjs/binaryjs or https://github.com/liamks/Delivery.js if you can run node.js on the server.