why do I need to poll message hub?

Question

I am looking at the node-js express example for message-hub on blue mix and am puzzled for why I need to poll message hub from my server. I thought that the whole idea behind a pub-sub model is that I don't have to (over) load my server polling the message service to find out if new messages exist for me to consume. In the example provided, lines 211-213 in the app.js file contain the following:

  // Set up an interval which will poll Message Hub for
  // new messages on the 'livechat' topic.
  produceInterval = setInterval(function() { ...},250);

This now has my server polling message hub every 250 mSec when what I want is to completely avoid a polling model and be notified by message hub when a message exists for me to consume.

Answer 1

in brief: Kafka achieves its scalability by using a consumer-pull model, not a server-push.

in detail: It's worth going through the Kafka documentation first. In particular, your question is answered here http://kafka.apache.org/documentation/#design_pull

HTH, Edo

Answer 2

With KafkaConsumer the poll() function means that your app is polling the client buffer, not necessarily polling across the network to the Kafka broker. Kafka clients often prefetch and cache data in this client side buffer for better performance, lower latency, and better network efficiency.

If you want an async callback style interface that "pushes" data to your app then it's very easy to wrap the polling interface and make it look like a push. Ultimately every push API has something hidden under the covers that is calling poll on a tcp socket.