Looking for clarity on Event Hubs vs Topics in Azu

Question

I've been learning about Event Hubs and just want to get confirmation or correction on my perspective on Event Hubs? I’m used to leveraging retries, poison messages, at least once delivery and so on for normal enterprise messaging solutions, which Azure Service Bus Queues and Topics give me. It seems that Event Hubs is intended to provide a different tool for very high scale where you have to give up a little of the more “enterprise” features for much higher scale.

Am I thinking about this correctly? Are there additional specifics I need to consider as well? I realize there could be some functional overlap with Event Hubs and Topics, but I'm just looking to get some clarity on how to think of using Event Hubs.

Answer 1

Correct!!

The fundamental difference between EventHubs and Topics is - TOPICS offer per-message semantics - but, EventHubs - offers Stream Semantics - you should not expect any per-message feature/semantics with EventHubs.

Any middle-tier providing per-message features comes with the processing overhead (the tax)!!

For Ex: Per message Duplicate detection, Receive confirmation per message (topics have a Message.Complete to ack Msg received) - are all Topic features. EventHubs narrows-down the feature set to provide a better low-latency/high-throughput solution.

To visualize features like at-least-once delivery (of per msg is not available in EventHubs) is to translate it to stream semantics - Read until a point in a Given eventHub partition and checkpoint and let your application which is consuming those events handle the at-least-once delivery.

Answer 2

If you have the choice it's almost always easier to write a system based around a full enterprise pubsub messaging system where you can mark single events as having been consumed, retry messages, and just about every other wonderful feature. If you've already accepted partitioning your message channel (which Azure Service Bus Topics appear to support) then you could in principle scale a more full featured messaging system to the degree you require. The issue is at what cost?

An Azure Service Bus Topic has a cost at high scale of approximately $0.20 per Million messages, Amazon SQS (somewhat similar) lists $0.50 per Million. If you host it yourself you'll likely need to set up a lot of RabbitMQ servers or even multiple clusters as you partition.

Azure Event Hub costs $0.028 per Million plus an amount per throughput unit, same for Amazon Kinesis. Apache Kafka has been benchmarked at 2 Million per second on 3 machines

At say 20,000 events per second sustained the difference between some Azure Topics and Azure Event Hub is in the range of a full time developer's salary. At 2 million per second sustained (which requires contacting MS), the difference is approaching $1M/month.

Basically use the partitioned stream|log / offset tracking systems when you either don't need all the useful features of a full messaging system, or when you don't need them enough to pay the ~10X premium. (Or can't use them because you can't scale the proper messaging system enough without heroic efforts).

Answer 3

i wrote a post a while ago about this topic with some support from Dan on the Service Bus Team. Hopefully this should clarify for you

http://microsoftintegration.guru/2015/03/03/azure-event-hubs-vs-azure-messaging/

Service Bus (Messaging)

For messaging it’s about one application telling one or more apps to DO SOMETHING or GIVE ME SOMETHING.

Event Hub (Eventing)

The alternative is that in eventing the applications are saying SOMETHING HAS HAPPENED.