Create multiple consumers in Kafka in command line

Question

I'm new in Kafka. When I was running the quick start example in command line, I found I can't create multiple consumers in command line.

Condition:

I built a topic named test with 3 partitions, and I also built a producer on this topic.

Then I wanted to create two different consumers sharing a same consumer-group named test1 on this topic.

I ran the command like below twice:

   bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --group test1

The first one worked but when I ran the second time the first one would disconnect and the second one worked.

So how can I create two or more consumers in a same consumer group in command line?

    WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)

Answer 1

use this:

--partition <Integer: partition>        The partition to consume from. 

Answer 2

1. Default, kafka-console-consumer.sh will create a random group.
2. If you want to specify the group name, you can:
   1. Add group.id=group_name to a local file filename
   2. Use --consumer.config filename option of kafka-console-consumer.sh to set group
3. You can check your groups at zookeeper's /consumers/ directory.

Refer: kafka/core/src/main/scala/kafka/tools/ConsoleConsumer.scala

Answer 3

Besides using --consumer.config option like the secfree's answer, you can also use

--consumer-property group.id=your_group

option to specify a group name without editing the config file.

Answer 4

You can use the below command to create the consumers in the group "test-consumer-group" to "test" topic:

bin/kafka-console-consumer.sh --bootstrap-server <brokerIP>:9092 --topic test --consumer-property group.id=test-consumer-group

Below command will list the consumer group configuration:

bin/kafka-consumer-groups.sh --bootstrap-server <brokerIP>:9092 --describe --group test-consumer-group

Eg:

GROUP || TOPIC || PARTITION || CURRENT-OFFSET || LOG-END-OFFSET || LAG      || OWNER
test-consumer-group || test || 0 || 10 || 10 || 0 || consumer-1_/10.210.223.170