logstash with log4j socketappender doesn't wor

Question

I am trying to stream my logs through the log4j socket appender and trying to retrieve the same through the logstash log4j input plugin. My logstash configuration looks like

input {
        log4j
        {
         type=>"socketlogs"
          port=>"1995"
        }
}

output {
        stdout {}
        file
        { path=>"socket.log"}

}

My log4j.xml looks like below

<appender name="logstashsocketappender" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=<logstash ip>/>
        <param name="Port" value="1995"/>
        <param name="ReconnectionDelay" value="60000"/>
        <param name="Threshold" value="INFO"/>   
    </appender>

<root>
        <level value="INFO"/>
        <appender-ref ref="logstashsocketappender"/>
    </root>

I have a logger.info statement in the code for testing, however I do not see those in stdout in the logstash machine. These are the software versions I am using

logstash - 1.4.2 log4j - 1.7.5

Let me know if I am missing something here.

Thanks for the help Gowri

Answer 1

I'm facing a similar issue using logstash-1.4.2 and log4j-1.2.17: Only some of my logged messages arrive at logstash. If I log bunches of 10 messages only 5 to 10 messages arrive at logstash. The longer I post messages (in bunches) the more messages arrive. (after some iterations all 10 messages arrive) - Maybe there is an issue at startup where some messages are lost?

Maybe you try to log some messages too, to delimit the problem...

If it would be a misconfiguration, you would get log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later. on stderr at the client side (sender).

It it seems to be a performance problem: If I wait 100ms between the messages of a bunch all messages arrive at logstash.

Answer 2

Have you tried to use GELF? The log4j internals use TCP. GELF uses UDP, so it does not slow down your application.

logstash.conf

input {
    gelf {
        port => 12201
    }
}

log4j.xml

<appender name="gelf" class="biz.paluch.logging.gelf.log4j.GelfLogAppender">
    <param name="Threshold" value="INFO" />
    <param name="Host" value="udp:localhost" />
    <param name="Port" value="12201" />
</appender>

See https://github.com/mp911de/logstash-gelf for more docs.