Best practice for loading a large text file using

Question

Currently I'm storing the results of the get in a string since the file I am opening is plain text file 3MB to 20MB in size.

Then I parse this string and modify it so that the end result can be outputted in html format.

I'm just looking for a sanity check to see if loading in this manner is the best way?

Also, is there a way to load a chunk of the target text file, parse the chunk, request another chunk, etc. Kind of like a music player buffering a song while playing the song.

Thank you

Answer 1

is there a way to load a chunk of the target text file, parse the chunk, request another chunk, etc.

To retrieve a portion (or chunk) of a resource, use the HTTP Range header. Most web servers honor this header correctly.

example HTTP Request:

GET /resource/url/path HTTP/1.1\r\n
User-agent: whatever\r\n
Host: www.example.com\r\n
Range: bytes=0-3200\r\n\r\n

Update:

With jQuery, v1.5 and later, you can tell jQuery to send additional HTTP headers for an individual ajax call, like this:

    $.ajax({type: "GET",
            url: url,
            headers : { "Range" : 'bytes=0-3200' },
            ....

If you don't want to modify each $.ajax() call, Stofke says you can use a "beforeSend" fn. Something like this:

  $.ajaxSetup({ beforeSend : function(xhr) {
        xhr.setRequestHeader("Range", "bytes=0-3200" );
  }});      

Using $.ajaxSetup(), you'd need to modify that range header for each additional call.

I don't know of a way to tell jQuery to load a resource chunkwise, automatically. I guess you'd have to do that yourself with setTimeout(), or something, in the success() of the ajax call. Each time you get a response, call setTimeout() and make the next ajax call. Your browser-resident Javascript will need to keep track of which portions of the file you have retrieved already, and which part (byte index) you want to get next.

Another way to do it would be to make the ajax calls only when your app is ready to do so. Rather than waiting for success() from the ajax call, just make an ajax call after you've finished processing the results of the first range retrieval.

Also: to support the arithmetic purposes, before doing the first GET, you can use the HEAD call to learn the length of the resource, which will tell you the max index in the range you can use.

Answer 2

Probably not what you need since I believe it still fetches the whole document and extracts a part out of it. But I'll add it anyway maybe it is useful.

From the jquery docs:

the .load() method, unlike $.get(), allows us to specify a portion of the remote document to be inserted. This is achieved with a special syntax for the url parameter. If one or more space characters are included in the string, the portion of the string following the first space is assumed to be a jQuery selector that determines the content to be loaded.

We could modify the example above to use only part of the document that is fetched:

$('#result').load('ajax/test.html #container');

When this method executes, it retrieves the content of ajax/test.html, but then jQuery parses the returned document to find the element with an ID of container. This element, along with its contents, is inserted into the element with an ID of result, and the rest of the retrieved document is discarded.