How to handle loading of LARGE JSON files

问题:

I've been working on a WebGL application that requires a tremendous amount of point data to draw to the screen. Currently, that point and surface data is stored on the webserver I am using, and ALL 120MB of JSON is being downloaded by the browser on page load. This takes well over a minute on my network, which is not optimal. I was wondering if anyone has any experience/tips about loading data this large. I've tried eliminating as much whitespace as possible, but that barely made a dent in file size.

Is there any way to either compress this file immensely, or otherwise a better way to download such a large amount of data? Any help would be great!

回答1:

JSON is incredibly redundant so it compresses well, which is then decompressed on the client.

JavaScript implementation of Gzip

Alternatively you could chunk the data up into 1 MB chunks to be sent over one at a time. Also the user probably can't interact with 120 MB of data at a time, so maybe implement some sort of level of detail system?

回答2:

If you control the web server sending the data, you could try enabling compression of json data.

This is done by adding the following in applicationhost.config (IIS 7):

<system.webServer>
    <urlCompression doDynamicCompression="true" />
    <httpCompression>
      <dynamicTypes>
        <add mimeType="application/json" enabled="true" />
        <add mimeType="application/json; charset=utf-8" enabled="true" />        
      </dynamicTypes>
    </httpCompression>
</system.webServer>

You would then need to restart the App pool for your app.

Source

回答3:

A few things you might consider:

1. Is your server compressing the file before sending?

2. Does this data change often? If it does not, you could set your expires header to a very long time, so the browser could keep it in cache. It wouldn't help on the first page access, but on subsequent ones the file wouldn't have to be loaded again.

3. Is there a lot of repeating stuff in your json file? For instance, if your object keys are long, you could replace them with shorter ones, send, and replace again in the browser. The benefits will not be that great if the file is compressed (see item 1) but depending on your file it might help a little.

4. Is all this data consumed by the browser at once? If it's not, you could try breaking it down into smaller pieces, and start processing the first parts while the others load.

But the most important: are you sure JSON is the right tool for this job? A general purpose compression tool can only go so far, but if you explore the particular characteristics of your data you might be able to achieve better results. If you give more details on the format you're using we may be able to help you more.

回答4:

I had an issue sort of like the one you had, so I've decide to use binary numbers instead of strings so when the user create request to my server I answer with a numbers.

For ex' lets say the user is a table in a restaurant so instead of sending a string like 'burger,orange juice, water,etc...' I can send the number 15 and translate into binary 8,4,2,1.

when the user ask for multiple thing, lets say 4 burgers it will be hard to follow so you can add two numbers of an array with the binary. I found it very useful and more secure.

If you decide doing it, I suggest on dev' mode use string when you deploy translate into binary.