Why __consumer_offsets topic in kafka is not sprea

Question

I have a 3 zk nodes cluster. And 7 kafka broker nodes.

So when I create any topic then I can set replica factor and no of partitions using command line arguements.

And these partitions are spread to all the 7 brokers.But there is one topic i.e. __consumer_offsets,it is created automatically and it is spreading to only 1 broker(id=0) with 1 replication.

I am aware that I could change the default no of replication factor in config file.

But, this topic is replicated to only one broker. So what is the parameter which could be changed to replicate the partitions to all the brokers.

Answer 1

You need to set the "offsets.topic.replication.factor" to the desired number of replicas in the broker config.

offsets.topic.replication.factor - "The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement."

Reference - http://kafka.apache.org/documentation/#brokerconfigs

Answer 2

You will need to use Kafka partition reassignment. This is unfortunately a semi manual process currently: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

Answer 3

It sounds really strange to me. I deployed a 3 Kafka brokers cluster and my __consumer_offsets topic has 50 partitions (the default) split across the 3 brokers and having 3 (the default) as replicas. What you are describing could happen when you start with a single Kafka broker, create consumers for reading topics so that the __consumer_offsets is automatically created and it will be just on the broker 0 (the only one). After that, you add new Kafka brokers: without doing anything the __consumer_offsets will still stay on broker 0; you need to use Kafka a manual partition reassignment for doing that as mentioned above.