What is the magic behind Lightstreamer?

Question

I'm gonna develop a framework for comet programming, and I can't use Web Sockets, or Server-Sent Events (because browser support really sucks). So, I need to keep the HTTP connection alive, and send chunked data back to the client.

However, problems show themselves as you get into the work:

1. Using XMLHttpRequest is not possible, due to the fact that IE doesn't give you xhr.responseText while the xhr.readyState is 3.
2. A hidden iframe can't be useful, because browser shows the loader while I send data back to the client.
3. I tried to send a JavaScript file back to the client, sending function execution commands each time, but browsers won't execute JavaScript till it's completely loaded.

However, when I look at Lightstreamer demo page, I see that it sends a JavaScript file back to the client little by little and in each step, it sends a call to the function and that function simply gets executed (I can't do this part). It seems that Lightstreamer uses AJAX, since the request simply shows up in Firebug's console tab, but it works like a charm in IE too.

I tried to use every HTTP header field they've set on their request, and no result. I also tried to use HTTP Post instead of HTTP Get, but still got no result.

I've read almost over 20 articles on how to implement comet, but none of'em appear to solve problems I have:

1. How to make it cross-browser?
2. How to get notified when new data is arrived from server (what event should I hook into)?
3. How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?

Can anyone please help? I think there should be a very little tip or trick that I don't know here to glue all the concepts together. Does anyone know what lightstreamer do to overcome these problems?

Answer 1

but browsers won't execute JavaScript till it's completely loaded.

Have you tried sending back code wrapped in <script> tags? For example, instead of:

<script type="text/javascript">
f(...data1...);
f(...data2...);

try

<script type="text/javascript">f(...data1...);</script>
<script type="text/javascript">f(...data2...);</script>

Answer 2

The methods you want is the streaming.

How to make it cross-browser?

Considering most browsers, there is no consistent way. You have to choose a proper transport according to the browser. Even worse, you have to rely on the browser sniffing to recognize which browser is being used, and the feature detection counts for nothing about this. You can use XDomainRequest for IE8+, XMLHttpRequest for non-IE and Iframe for IE 6+. Avoid iframe transport if possible.

How to get notified when new data is arrived from server (what event should I hook into)?

This varies according to the transport being used. For example, XDomainRequest fires progress event, XMLHttpRequest fires readystatechange event when chunk is arrived except Opera and IE.

How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?

I don't know this issue with iframe, but still occurs in WebKit based browsers such as Chrome and Safari with XMLHttpRequest. The only way to avoid this is to connect after the onload event of window, but, in case of Safari, this does not work.

There are some issues you have to consider besides the above questions.

1. Event-driven server - The server should be able to process asynchronously.
2. Transport requirements - The server behaves differently for required transport.
3. Stream format - If the server is going to send big message or multiple messages in a single chunk, a single chunk does not mean a single data. It could be fragment of a single data or concatenation of multiple data. To recognize what is data, the response should be formatted.
4. Error handling - Iframe transport does not provide any evidence for disconnection.
5. ...

Last but not least, to implement streaming is pretty tiresome than it looks unlike with long polling. I recommend you use solid framework for doing that such as socketio, sockjs and jquery socket which I've created and managed.

Good luck.

Answer 3

SockJS author here.

• How to make it cross-browser?

This is hard, expect to spend a few months on getting streaming transports on opera and IE.

• How to get notified when new data is arrived from server (what event should I hook into)?

There are various techniques, depending on a particular browser. For a good intro take a look at different fallback protocols supported by Socket.IO and SockJS.

• How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?

Again, there are browser-specific tricks. One is to delay loading AJAX after onload event. Other is to bind-and-unbind an iframe from DOM. ETC. If you still feel interested read SockJS or Socket.io code.

Can anyone please help? I think there should be a very little tip or trick that I don't know here to glue all the concepts together. Does anyone know what lightstreamer do to overcome these problems?

Basically, unless you have a very strong reason to, don't reinvent the wheel. Use SockJS, Socket.io, faye, or any other of dozens projects that do solve this problem already.

Answer 4

The best option in your case would be to use JSONP + Long Pulling on server side. You just have to remember to reconnect any time connection drops (times out) or you receive response.

Example code in jquery:

function myJSONP(){
    $.getScript(url, {
        success: function(){
            myJSONP(); //re-connect
        }, 
        failure: function(){
            myJSONP(); //re-connect
        }
    })
} 

Obviously your response from server has to be javascript code that will call on of your global functions.

Alternatively you can use some jquery JSONP plugin.

Or take a look on this project http://www.meteor.com/ (really cool, but didn't try it)