Exception during topic deletion when Kafka is host

Question

I host Kafka in Docker in Windows. Wurstmeister/Kafka docker image is used. Kafka data is stored in local Windows folder for persistency. Windows folder is mapped to Kafka docker image via Docker volumes. I can create topics, publish and consume messages. However when I try to delete topic I receive the following error:

 Error while deleting test-0 in dir /var/lib/kafka. (kafka.server.LogDirFailureChannel)
 java.io.IOException: Failed to rename log directory from /var/lib/kafka/test-0 to /var/lib/kafka/test-0.a81ff9700e4e4c3e8b20c6d949971b64-delete
 at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
 at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
 at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
 at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
 at kafka.cluster.Partition.delete(Partition.scala:173)
 at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:341)
 at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(ReplicaManager.scala:373)
 at scala.collection.Iterator.foreach(Iterator.scala:929)
 at scala.collection.Iterator.foreach$(Iterator.scala:929)
 at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
 at scala.collection.IterableLike.foreach(IterableLike.scala:71)
 at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
 at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
 at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:371)
 at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:190)
 at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
 at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
 at java.lang.Thread.run(Thread.java:748)

Could somebody help me to cope with this issue?

UPD: Below you can find contents of docker-compose file that I use to run Kafka:

version: '3'
services:
  zookeeper:
    image: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
    volumes:
      - ./zookeeper_data:/data
      - ./zookeeper_datalog:/datalog
  kafka:
    depends_on: 
      - zookeeper
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_LOG_DIRS: /var/lib/kafka
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_BROKER_ID: 1
    volumes:
      - ./kafka_logs:/var/lib/kafka

Answer 1

This issue still exists in Windows for Kafka ver 1.1.0 (kafka_2.12-1.1.0) when I try to delete the topic.

The topic gets marked for deletion and the Kafka server fails with java.nio.file.AccessDeniedException when trying to rename the logs directory 'test-0'

Deleting the whole test-0 logs folder does not help. Reinstalling the Kafka server does not help either - even after reinstalling, the info about the topic marked for deletion remains.

Took me a couple of hours to figure out that the info about the topic sits in the Zookeeper - in one of the log files!

Solution

Stop the Zookeeper process. Go to your Zookeeper logs folder zookeeper-3.x.x\bin\zookeeper-3.x.xdata\version-2\ and delete the latest log.xx files. Restart Zookeper. Restart Kafka server.

Answer 2

This solved the problem
1.Stop both Kafka and Zookeeper processes.
2.Delete all old log directories.
3.Change the log.dir to point to new directory in server.properties
4.Change the dataDir to point to new directory in zookeeper.properties

Then restart the Zookeeper and Kafka server:

C:\kafka_2.12-2.4.0> zookeeper-server-start.bat .\config\zookeeper.properties
C:\kafka_2.12-2.4.0> kafka-server-start.bat .\config\server.properties

Answer 3

1, Change the log.dir with a new name in server.properties ->Kafka/config

log.dir=C:/Programs/kafka/kafka_2.12-2.3.0/kafka-test-logs

2, Remove the old log folder from C:/Programs/kafka/kafka_2.12-2.3.0/

3, Remove all logs and snapshot from C:\Programs\zookeeper\apache-zookeeper-3.5.5-bin\data or delete the data folder where your log is stored

Additionally, I had an error when starting the consumer(Leader Not Available Kafka in Console Producer),

I added,

port = 9092 advertised.host.name = localhost

to the server.properties

Now am able to publish and consume messages

Answer 4

Delete version-2 from Zookeeper logs folder.
Delete all things in Kafka-logs folder.

Then restart the Zookeeper and Kafka server:

• zookeeper-server-start.bat D:\kafka_2.11-2.4.1\config\zookeeper.properties
• kafka-server-start.bat D:\kafka_2.11-2.4.1\config\server.properties

Answer 5

Deletion of topic fails due to Java's File.rename function. It works differently in some cases in Windows environment (for example if file is in use). Kafka developers already changed this function to Utils.atomicMoveWithFallback (see this issue for details), but seems it was not included into Kafka 2.11-0.11.0. So you need to work with Kafka version that has this fix. Hope this will helps.