Zookeeper usage on Kafka 0.9.0

问题:

Can someone tell me in kafka 0.9.0 (saving the offsets in kafka and using the new consumer) what are the functions that zookeeper perform that may require lot of resources? will this resources fluctuate a lot in a data spike? or only if something crashes in the cluster?

回答1:

As you've said, ZooKeeper is no longer used by the producer or consumer in 0.9.0. However, Kafka stores much of its metadata in ZooKeeper -- topics, partitions, replicas, the ISR, brokers, etc. A single produce or consume does not interact with ZooKeeper. Instead, ZooKeeper is interacted with in "broader" operations: adding a new broker, adding a new topic, recovering from a failed broker, recovering from a failed controller, etc. Also, brokers heartbeat with ZooKeeper, allowing ZooKeeper to know if a broker fails.

In general, in 0.9.0, ZooKeeper will only be a performance bottleneck if you have a massive cluster, a massive number of partitions, or very "small" ZooKeeper nodes.