How to copy a topic from a kafka cluster to anothe

Question

One way to do that as the Kafka documentation shows is through kafka.tools.MirrorMaker which can do that trick. However, I need to copy a topic (say test with 1 partition) (its content and meta data) from a production environment to a development environment where connectivity is not there. I could do simple file transfer between environments though. My question: if I move the *.log and .index from the folder test-0 to the destination Kafka cluster, is that good enough? Or there is more that I need to do like meta data and ZooKeeper-related data that I need to move too?

Answer 1

Just copying the log and indexes will not suffice - kafka stores offsets and topic meta data in zookeeper. MirrorMaker is actually a quite simple tool, it spawns consumers to the source topic as well as producers to the target topic and runs until all consumers consumed the source queue. You can't find a simpler process to migrate a topic.

Answer 2

What worked for me in your scenario was the following sequence of actions:

1. Create the topic in Kafka where you will later insert your files (with 1 partition and 1 replica and an appropriate retention.ms config so that Kafka doesn't delete your presumably outdated segments).
2. Stop your Kafka and Zookeeper.
3. Find the location of the files of the 0-partition you created in Kafka in step 1 (it will be something like kafka-logs-<hash>/<your-topic>-0).
4. In this folder, remove the existing files and copy your files to it.
5. Start Kafka and Zookeeper.

This also works if your Kafka is run from docker-compose (but you'll have to set up an appropriate volume, of course).