Kafka: how to remove broker from Replica Set

Question

I have a partition with the following Replicas:

Topic: topicname    Partition: 10   Leader: 1   Replicas: 1,2,4,3   Isr: 1,2,3

Where Replica 4 is a non-existent broker. I accidentally added this broker into the replica set as a typo.

I want to remove 4 from the Replica set. but after running kafka-reassign-partitions.sh, the reassignment to remove Replica #4 never finishes.

kafka-reassign-partitions.sh --zookeeper myzookeeperhost:2181 --reassignment-json-file remove4.txt --execute 

Where remove4.txt looks like

{ "partitions": [
     { "topic": "topicname", "partition": 2, "replicas": [1,2,3] }
], "version": 1 }

The reassignment is stuck:

kafka-reassign-partitions.sh --zookeeper myzookeeperhost:2181 --reassignment-json-file remove4.txt --verify
Status of partition reassignment:
Reassignment of partition [topicname,10] is still in progress

I checked the controller log, it looks like the reassignment command was picked up, but nothing happens afterwards:

[2017-08-01 06:46:07,653] DEBUG [PartitionsReassignedListener on 101 (the controller broker)]: Partitions reassigned listener fired for path /admin/reassign_partitions. Record partitions to be reassigned {"version":1,"partitions":[{"topic":"topicname","partition":10,"replicas":[1,2,3]}]} (kafka.controller.PartitionsReassignedListener)

Any ideas on what I'm doing wrong? How to I remove broker #4 from the replica set? update: I'm running kafka 10

Answer 1

I was able to solve this issue by spinning up a new broker with a broker-id matching the one that was added (in your case 4).

The Kafka Quickstart guide shows you how to spin up a broker with a specific id. Once you have your node up with id 4, run:

./bin/kafka-topics.sh --zookeeper localhost:2181 --topic badbrokertest --describe

You should see that all the replicas are in the isr column like so:

Topic:badbrokertest PartitionCount:3    ReplicationFactor:3 Configs:
Topic: badbrokertest    Partition: 0    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3
Topic: badbrokertest    Partition: 1    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3  
Topic: badbrokertest    Partition: 2    Leader: 1   Replicas: 1,2,3,4   Isr: 1,2,3,4

Now you can reassign your partitions!

./bin/kafka-reassign-partitions.sh --reassignment-json-file badbroker2.json  --zookeeper localhost:2181

Where badbroker2.json looks like:

{
    "version":1,
    "partitions":[
        {"topic":"badbrokertest","partition":0,"replicas":[1,2,3]},
        {"topic":"badbrokertest","partition":1,"replicas":[1,2,3]},
        {"topic":"badbrokertest","partition":2,"replicas":[1,2,3]}
    ]
}

So in short, once you've sync'd all your replicas by adding the missing broker you can remove the unneeded broker.

If you're working with several servers, be sure to set the listeners field in the config to make your temporary broker available to the others brokers. The Quickstart guide doesn't consider that case.

listeners=PLAINTEXT://10.9.1.42:9093