Is there any reason to use RabbitMQ over Kafka?

Question

I've been asked to evaluate RabbitMQ instead of Kafka but found it hard to find a reason that it's doing something better than Kafka. Does anyone knows if is really better in throughput, durability, latency, or ease-of-use?

Answer 1

RabbitMQ is a solid, general purpose message broker that supports several protocols such as AMQP, MQTT, STOMP etc. It can handle high-throughput and common use cases for it is to handle background jobs or as message broker between microservices. Kafka is a message bus optimized for high-ingress data streams and replay.

Kafka can be seen as a durable message broker where applications can process and re-process streamed data on disk. Kafka has a very simple routing approach. RabbitMQ has better option if you need to route your messages in complex ways to your consumers. Use Kafka if you need to supporting batch consumers that could be offline, or consumers that want messages at low latency. 

RabbitMQ will keep all states about consumed/acknowledged/unacknowledged messages while Kafka doesn't, it assumes the consumer keep tracks of what's been consumed and not. RabbitMQ's queues are fastest when they're empty, while Kafka retain large amounts of data with very little overhead - Kafka is designed for holding and distributing large volumes of messages. (If you plan to have very long queues in RabbitMQ you could have a look at lazy queues.)

Kafka is built from the ground up with horizontal scaling (scale by adding more machines) in mind, while RabbitMQ is mostly designed for vertical scaling (scale by adding more power ).

RabbitMQ has a user-friendly interface that let you monitor and handle your RabbitMQ server from a web browser. Among other things queues, connections, channels, exchanges, users and user permissions can be handled - created, deleted and listed in the browser and you can monitor message rates and send/receive messages manually. Kafka manager is yet not as developed as RabbitMQ Management interface. I would say that it's easier/gets faster to get a good understanding about RabbitMQ.

More reading and some comparison data can be found here: https://www.cloudkarafka.com/blog/2016-12-05-apachekafka-vs-rabbitmq.html

Also recommending the industry paper: "Kafka versus RabbitMQ: A comparative study of two industry reference publish/subscribe implementations": http://dl.acm.org/citation.cfm?id=3093908

I do work at a company providing both Apache Kafka and RabbitMQ as a Service.

Answer 2

I hear this question every week... While RabbitMQ (like IBM MQ or JMS or other messaging solutions in general) is used for traditional messaging, Apache Kafka is used as streaming platform (messaging + distributed storage + processing of data). Both are built for different use cases.

You can use Kafka for "traditional messaging", but not use MQ for Kafka-specific scenarios.

The article “Apache Kafka vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? (https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/)” discusses why Kafka is not competitive but complementary to integration and messaging solutions (including RabbitMQ) and how to integrate both.