How to create a streaming API with NodeJS

Question

How would you go to create a streaming API with Node? just like the Twitter streaming API.

What I want to do ultimately is get the first update from the FriendFeed api, and stream when a new one becomes available (if the id is different), and later on expose it as a web service so I can use it with WebSockets on my website :).

So far I have this:

var sys = require('sys'),
    http = require('http');

var ff = http.createClient(80, 'friendfeed-api.com');
var request = ff.request('GET', '/v2/feed/igorgue?num=1', 
                         {'host': 'friendfeed-api.com'});

request.addListener('response', function (response) {
    response.setEncoding('utf8'); // this is *very* important!
    response.addListener('data', function (chunk) {
        var data = JSON.parse(chunk);
        sys.puts(data.entries[0].body);
    });
});
request.end();

Which only gets the data from FriendFeed, creating the Http server with node is easy but it can't return a stream (or I haven't yet found out how).

Answer 1

You would want to set up a system that keeps track of incoming requests and stores their response objects. Then when it's time to stream a new event from FriendFeed, iterate through their response objects and responses[i].write('something') out to them.

Check out LearnBoost's Socket.IO-Node, you may even just be able to use that project as your framework and not have to code it yourself.

From the Socket.IO-Node example app (for chat):

io.listen(server, {

    onClientConnect: function(client){
        client.send(json({ buffer: buffer }));
        client.broadcast(json({ announcement: client.sessionId + ' connected' }));
    },

    onClientDisconnect: function(client){
        client.broadcast(json({ announcement: client.sessionId + ' disconnected' }));
    },

    onClientMessage: function(message, client){
        var msg = { message: [client.sessionId, message] };
        buffer.push(msg);
        if (buffer.length > 15) buffer.shift();
        client.broadcast(json(msg));
    }

});