Converting Uint8Array crashing browser for large f

2019-05-25 06:48发布

问题:

Have an app where there is an input of type "file". The following methods grab the file, then prep it to be sent to the server via AJAX.

private StartUpload = (files) => {
    if (files && files.length === 1) {
        this.GetFileProperties(files[0])
            .done((properties: IFileProperties) => {
                $('input[type=file]').val("");
                if (this._compatibleTypes.indexOf(properties.Extension) >= 0) {
                    var base64 = this.ArrayBufferToBase64(properties.ArrayBuffer);

                    this.DoFileUpload(base64, properties.Extension).always(() => {
                        this.ShowDialogMessage('edit_document_upload_complete', 'edit_document_upload_complete');
                    });
                } else {
                    this.ShowDialogMessage('edit_document_upload_incompatible', 'edit_document_upload_compatible_types', this._compatibleTypes);
                }
            });
    } else {
        this.ShowDialogMessage('edit_document_upload_one_file', 'edit_document_upload_one_file_msg');
    }
};

private ArrayBufferToBase64(buffer): any {
    var binary = '';
    var bytes = new Uint8Array(buffer);
    for (var xx = 0, len = bytes.byteLength; xx < len; xx++) {
        binary += String.fromCharCode(bytes[xx]);
    }
    return window.btoa(binary);
}

private DoFileUpload = (base64, extension) => {
    this.IsLoading(true);
    var dfd = $.Deferred();

    var data = {
        data: base64
    };

    UpdateFormDigest((<any>window)._spPageContextInfo.webServerRelativeUrl, (<any>window)._spFormDigestRefreshInterval);

    var methodUrl = "_vti_bin/viewfile/FileInformation.asmx/AddScannedItemAlt";

    $.ajax({
        headers: {
            "X-RequestDigest": $("#__REQUESTDIGEST").val()
        },
        url: methodUrl,
        contentType: "application/json",
        data: JSON.stringify(data),
        dataType: 'json',
        type: "POST",
        success: (response) => {
            // do stuff
        },
        error: (e) => {
            // do stuff
        }
    });

    return dfd;
};

This works perfectly in the vast majority of cases. However, when the file size is large (say 200MB+) it kills the browser.

  • Chrome shows a blackish-grey page with the "aw snap" message and basically dies.

  • IE shows an "Out of Memory" console error but continues to work.

  • FF shows an "Unresponsive script" warning. Choosing "don't show me again" lets it run until an "out of memory" console error shows up.

This is where it dies:

for (var xx = 0, len = bytes.byteLength; xx < len; xx++) {
    binary += String.fromCharCode(bytes[xx]);
}

Wrapping a try/catch around this does nothing and no error is caught.

I can step into the loop without a crash, but stepping through every iteration is tough since len = 210164805. For this I tried to add console.log(xx) to the loop and let it fly - but the browser crashes before anything shows up in the log.

Is there some limit to the size a string can be that could be causing the browser to crash once exceeded?

Thanks

回答1:

You need to do this in an asynchronous way by breaking up the code either in blocks or time segments.

This means your code will need to use callback, but otherwise it's straight forward -

Example

var bytes = new Uint8Array(256*1024*1024);  // 256 mb buffer

convert(bytes, function(str) {              // invoke the process with a callback defined
   alert("Done!");
});

function convert(bytes, callback) {

  var binary = "", blockSize = 2*1024*1024, // 2 mb block
      block = blockSize,                    // block segment
      xx = 0, len = bytes.byteLength;
  
  (function _loop() {
    while(xx < len && --block > 0) {        // copy until block segment = 0
      binary += String.fromCharCode(bytes[xx++]);
    }
    
    if (xx < len) {                         // more data to copy?
      block = blockSize;                    // reinit new block segment
      binary = "";                          // for demo to avoid out-of-memory
      setTimeout(_loop, 10);                // KEY: async wait
      
      // update a progress bar so we can see something going on:
      document.querySelector("div").style.width = (xx / len) * 100 + "%";
    }
    else callback(binary);                  // if done, invoke callback
  })();                                     // selv-invoke loop
}
html, body {width:100%;margin:0;overflow:hidden}
div {background:#4288F7;height:10px}
<div></div>

Using large buffers converted to string will possibly make the client run out of memory. A buffer of 200mb converted to string will add 2 x 200mb as strings are stored as UTF-16 (ie. 2 bytes per char), so here we use 600 mb out of the box.

It depends on browser and how it deals with memory allocations as well as the system of course. The browser will try to protect the computer against malevolent scripts which would attempt to fill up the memory for example.

You should be able to stay in ArrayBuffer and send that to server.