Kafka consumer with new API not working

Question

I found something very weird with Kafka.

I have a producer with 3 brokers :

bin/kafka-console-producer.sh --broker-list localhost:9093, localhost:9094, localhost:9095 --topic topic

Then I try to run a consumer with the new API :

bin/kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094,localhost:9095 --topic topic --from-beginning

I got nothing ! BUT if I use the old API :

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic topic

I got my messages !

What is wrong with me ?

PS : I am using Kafka 10

Answer 1

After fighting a while with same problem. Specify --partition and console consumer new API works (but hangs..). I have CDH 5.12 + Kafka 0.11 (from parcel).

UPD:

Also find out that Kafka 0.11 (versioned as 3.0.0 in CDH parclel) does not work correctly with consuming messages. After downgrading to Kafka 0.10 it become OK. --partition does not need any more.

Answer 2

I eventually resolved my problem thanks to this similar post : Kafka bootstrap-servers vs zookeeper in kafka-console-consumer

I believe it is a bug / wrong configuration of mine leading to a problem with zookeeper and kafka.

SOLUTION :

First be sure to have enable topic deleting in server.properties files of your brokers :

# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true

Then delete the topic :

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic myTopic

Remove all the /tmp/log.dir directories of your brokers.

EDIT : I faced again the problem and I had to remove also the log files of zookeeper in /tmp/zookeeper/version-2/.

Finally delete the topic in /brokers/topics in zookeeper as follow :

$ kafka/bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

rmr /broker/topics/mytopic

And restart your brokers and create your topic again.

Answer 3

I had the same problem, but I was using a single broker instance for my "cluster", and I was getting this error:

/var/log/messages

[2018-04-04 22:29:39,854] ERROR [KafkaApi-20] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis) 

I just added in my server configuration file the setting offsets.topic.replication.factor=1 and restarted. It started to work fine.