Kafka Log4j appender not sending messages

Question

i am quite new ot apache Kafka and log4j. i am trying to send my log messages into Kafka. Here is my log4j properties file

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n

log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.BrokerList=localhost:9092
log4j.appender.KAFKA.Topic=kfkLogs


log4j.appender.KAFKA.SerializerClass=kafka.producer.DefaultStringEncoder
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n

log4j.logger.logGen=DEBUG, KAFKA

however, i am not able to receive any messages in my consumer. I have tested the consumer with some other producer code and it works fine.

also, i get this warning

log4j:WARN No such property [serializerClass] in kafka.producer.KafkaLog4jAppender.

Edit

and here is the code that generates my log messages

    package logGen;

import org.apache.log4j.Logger;

public class TestLog4j {

    static Logger log = Logger.getLogger(TestLog4j.class.getName());

    public static void main(String[] args) {

        log.debug("Debug message");

        log.info("Info message");

        log.error("Error Message");

        log.fatal("Fatal Message");

        log.warn("Warn Message");

        log.trace("Trace Message");
    }

}

Also, if i write the log messages to a file by using something like

log4j.appender.KAFKA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.KAFKA.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.KAFKA.File=logs/server.log

i can see the log messages in the server.log file

thanks for the suggestions everyone. i think the weird behavior that i see might be related to my kafka set up. here are the contents of my server.properties file which i use to start up my kafka server. can you see anything odd about it ?

broker.id=0

port=9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/Users/xyz/kafka/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

log.cleaner.enable=false

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000

delete.topic.enable=true

Answer 1

I have took a look at the source code of KafkaLog4jAppender.scala and here are the valid and exhaustive properties for Kafka log4j appender as of version 0.8.2.1 : topic, brokerList, compressionType, requiredNumAcks, syncSend.

The log4j.properties that worked for me is below :

log4j.rootLogger=ERROR, stdout

log4j.logger.logGen=DEBUG, KAFKA

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n

log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.topic=LOG
log4j.appender.KAFKA.brokerList=localhost:9092
log4j.appender.KAFKA.compressionType=none
log4j.appender.KAFKA.requiredNumAcks=0
log4j.appender.KAFKA.syncSend=true
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n

Answer 2

log4j.appender.KAFKA.SerializerClass=kafka.producer.DefaultStringEncoder

could you try

log4j.appender.KAFKA.Serializer=kafka.producer.DefaultStringEncoder

Instead?

I believe the async mode of sending msgs does so by batching, hence the delay, have you you tried sending using sync?

Answer 3

You need to add KAFKA to your log4j.rootLogger like this:

log4j.rootLogger=INFO, stdout, KAFKA

This will add the KAFKA appender to your rootLogger.

Answer 4

i had to specify

log4j.appender.KAFKA.producer.type=async

log4j.logger.logGen.TestLog4j=TRACE, KAFKA

and that seems to work. however i experience a delay of 10-30 seconds. Specifically, if i publish right now and can see the messages in teh consumer, then the next time i publish has to be about 30 secs later else i dont see anything in my consumer. any ideas on why this might be happening ? maybe its an eclispe issue ?