Getting the list of Brokers Dynamically

Question

I am new to Kafka. I wrote my first message producer like this

private Properties kafkaProps = new Properties();
kafkaProps.put("bootstrap.servers", "broker1:9092,broker2:9092");
kafkaProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
kafkaProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
producer = new KafkaProducer<String, String>(kafkaProps);

is it possible that I bootstrap to Kafka and ask it what brokers are present and then put them in the properties? Otherwise there will always be a problem that the config of the code is out of sync with the actual state of the cluster.

Answer 1

Normally, kafka brokers will be managed by a zookeeper, so you can access the zookeeper instance to gather information about kafka brokers.

You can use, for example, a org.apache.storm.shade.org.apache.zookeeper.ZooKeeper instance to connect to Zookeeper, and run its getChildren(pathToBrokerIds, false) method which returns a list of kafka brokers' ids. Then you can run zookeeper's getData(..) method with each id as an argument, and get the info for that broker, including host and port.