Kafka Producer - By default supports Multithreadin

Question

I am newbie to kafka. I have created sample kafka sync producer and consumergroup programs using kafka_2.9.2-0.8.1.1. So My question is, do I need to add multithreading code to producer (like consumergroup class has) to support huge number of requests? I read producer send method is thread safe. So kafka producer will take care of multithreading concepts internally or developer has to code explicitly?

Any help would be highly appreciated.

Thanks, Cdhar

Answer 1

There are two types of producers available with Kafka. (1) SyncProducer (2) AsyncProducer. If you set the producer.type configuration as async it will uses the AsyncProducers. By default it uses the Synchronous producer class.

Once running in async mode it creates a separate AsyncProducer instance per broker.And each of these AsyncProducer instances maintains its own internal background thread for sending the messages. These are called ProducerSendThread.

So there is one thread running per broker and your parallelism is based on the number of brokers available in the cluster. So adding new brokers in the cluster should provide you the flexibilities to increase the level of parallelism while producing data using Kafka.But remember adding a new broker to your cluster should be considered taking other paramaters also into consideration.