Kafka Consumers are balanced across topics

Question

I am creating 6 consumers with same group-id. All the consumers are subscribing to 2 topics with 3 partitions each. Since there are 6 consumers and 6 partitions across 2 topics I am expecting all the consumers to be used. But I don't see all the consumers being used, is there a way I can force it to re balance ?

I am using kafka 0.10.2.0

Answer 1

Assigning partitions to consumers in the same consumer group is not done across topics. Following what is happening to you ... You have t1_p0, t1_p1 and t1_p2 (partitions on topic1) and t2_p0, t2_p1 and t2_p2 (partitions on topic2). Then you have a consumer group with 6 consumers c1, c2, ... c6. When they subscribe to topic1, it can happen that c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. When the same consumers subscribe to topic2, it can happen the same: c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. I know that you would like having c4, c5 and c6 getting these partitions from topic 2 but as I said partitions assignment doesn't happen across topics but for partitions in the same topic. In this scenario you have c4, c5 and c6 in idle and not used.