I am trying to understand the differences between something like Kafka and something like Camel. To my understanding Camel would provide much more abstraction for developers without having to worry about changing protocols/systems to some extent. How would Kafka not be able to handle most of what Camel can do now? I am reading through the documentation and it seems like Kafka has been updated/upgraded enough to slightly break away from being a message broker only. I guess my question would really come down to how does Kafka compare to Camel in regards to future proofing systems and where does Kafka fall short of Camel? I am under the impression that Kafka doesn't scale as well as a system grows.
Edit: This is strictly based around messages.The documentation surrounding Camel makes it very clear that it's based around Enterprise Integration Patterns, but the deeper I dive into Kafka documentation the same patterns can be implemented. Am I missing something?
Apache Kafka : Is a streaming processing platform. It is based on massively scalable publish subscribe message queue architecture. There are many other platforms which are based on JMS publish subscribe model, which could do the same(with some exceptions). Some of the most popular are Apache-Activemq, RabbitMq
Apache Camel : Is a message oriented middleware. It has implemented almost all the Enterprise Integration Patterns.
You can use Apache Camel with Apache Kafka. Or you can use Apache Kafka without Apache Camel also.
They are two totally different things.
Think about Camel as an interface definition tool where you can define endpoints or channels where messages fly in. But they are abstract. Compare Camel with Spring Integration for instance.
Kafka can provide those messages, so it can implement those abstract channels or endpoints. But so can ActiveMQ and others.
Kafka is a message broker. It is comparable with other message brokers like ActiveMQ, RabbitMQ, Azure Service Bus etc. Camel is an integration middleware. It is more comparable to Apache ServiceMix.