kafka Old consumer_offsets are not getting deleted

Question

Kafka Version 0.10.1.1

The brokers are having all the log cleanup/retention defaults:

log.cleanup.policy=delete
log.retention.hours=168 (7 days)
offsets.retention.minutes=1440 (1 day)
log.index.size.max.bytes=10485760 (10M)

But I can see the __consumer_offsets from Jul 6:

[root@npe-tfb-kfk-w2b-b3 __consumer_offsets-13]# ls -ltr -h
total 71M
-rw-r-----. 1 root root 70M Jul  6 08:31 00000000000081206436.log
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.index
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.timeindex

In Production I can see consumer_offsets .log file from Jul 16.

How is this possible? If I manually delete the old .log, .index and .timeindex files, is that going to impact the stability of the cluster.

Answer 1

The date discrepancy here is likely because of a restart with an unclean shutdown. The index files are rebuilt on such cases. As far as your offsets topic goes, you shouldn't manually deleted it unless you know for sure that's what you want. It will potentially cause a loss of state for your consumers. The topic is compacted and you don't have much data in it at all. I expect that you have previously deleted segments before because I don't see log segment 0 there. Just make sure your log cleaner is still functional and you will be ok. If you are worried about disk space because of a 70MB log segment, then you need more disk. Kafka disks should be 100s of GB in size.