log4j2 - Syslog appender and PatternLayout

Question

I need to log events into the syslog. I use lo4j2 and the syslog appender. My appenders block in log4j2.xml looks like this:

<appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <Syslog name="syslog" host="localhost" port="514" protocol="UDP" charset="ISO-8859-1">
        </Syslog>
        <RollingFile name="AppLog" fileName="/var/log/app.log"
                     filePattern="/var/log/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
            <Policies>
                <TimeBasedTriggeringPolicy/>
            </Policies>
        </RollingFile>          
    </appenders>

As you can see I have a Console appender and RollingFile appender with a specific PatternLayout. I want to use the same PatternLayout for the Syslog appender. However, the log messages in the syslog seem to always use a predefined layout. I tried to do the following:

<Syslog name="syslog" host="localhost" port="514" protocol="UDP" charset="ISO-8859-1">
    <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Syslog>

But this does not have any effect. the syslog messages still have the same predfined format.

How can I determine the format of my log messages that go into the syslog?

Answer 1

As mentioned in this log4j2 bug report, the developers of log4j2 coded the SyslogAppender as a SocketAppender hardwired to a SyslogLayout

because it is intended to conform to either the original syslog format or RFC 5424. No other Layout should be permitted.

They unfortunately did not realize that the RFC 5424 specifications do not enforce any particular format for the message contained in the log, that in the Log4j2 implementation is only the %m portion of the log.

To solve this issue, a solution (suggested in the same bug report) is to reproduce the syslog format using a PatternLayout inside a SocketAppender, like so

<Socket name="SYSLOG" host="localhost" port="514" protocol="UDP">
  <PatternLayout
    pattern="&lt;1&gt;%d{MMM dd HH:mm:ss} ${hostName} appName: {
      &quot;host&quot;:&quot;${hostName}&quot;,
      &quot;thread&quot;:&quot;%t&quot;,
      &quot;level&quot;:&quot;%p&quot;,
      &quot;logger&quot;:&quot;%c{1}&quot;,
      &quot;line&quot;:%L,
      &quot;message&quot;:&quot;%enc{%m}&quot;,
      &quot;exception&quot;:&quot;%exception&quot;
      }%n"
  />
</Socket>

This will write well-formatted RFC5424 logs to local 514 port through UDP. Following is a sample log output:

Sep 14 10:40:50 app-hostname app-name: { "host":"host-name-01", "thread":"http-nio-8080-exec-4", "level":"DEBUG", "logger":"ExecuteTimeInterceptor", "line":52, "message":"GET /health 200 served in 3", "exception":"" }

Answer 2

You can use SocketAppender and PatternLayout to format syslog (syslog-ng) messages.
To support dynamic Severities with a fixed Facility (e.g.: 'user-level messages' - see RFC5424) the pattern should look like this:

<Socket name="SYSLOG" host="${env:INTERFACE}" port="514" protocol="UDP">
   <PatternLayout pattern="&lt;%level{TRACE=15, DEBUG=15, INFO=14, WARN=12, ERROR=11, Fatal=11,&gt;%replace{${env:APPLICATION_NAME}}{\r}{}[%X{PID}] %t(%T) %c{10} - %m%n"/>
</Socket>

To calculate the Priority value (PRIVAL) for Facility 'user-level message' and Severity 'informational messages' - see RFC5424) the following example might help:

Syslog:          Facility  | Severity
Numerical Code:      1          6
Bin:             0 0 0 0 1 |  1 1 0
Dec:                 8     +    6    =  14

log4j2syslog-ngsocketappenderpatternlayout

Answer 3

I used the config posted by butcher82, but had to change it a bit to produce the result I needed.

What I got in the end is a message with the correct priority, timestamp (without leading zeros for days), host and a message part. The mapping between syslog and log4J level is used as defined in org.apache.log4j.Level and the facility is set to 1 (user-level messages), to simplify the priority calculation.

This pattern should be compatible with RFC-3164:

<Socket name="SysLogAppender" host="localhost" port="514" protocol="UDP">
    <PatternLayout pattern="&lt;%level{TRACE=7, DEBUG=7, INFO=6, WARN=4, ERROR=3, Fatal=0}&gt;%d{MMM d hh:mm:ss} ${hostName} %m%n"/>
</Socket>

Below is the produced output:

<3>Dec 15 09:59:16 foo.bar.hostname this is a test message

Note: One might add an application name or pid, after the hostname.

Answer 4

You can use add additional elements to an RFC5424 formatted SyslogAppender message using the LoggerFields tag like this:

<LoggerFields>
  <KeyValuePair key="thread" value="%t"/>
  <KeyValuePair key="priority" value="%p"/>
  <KeyValuePair key="category" value="%c"/>
  <KeyValuePair key="exception" value="%ex"/>
</LoggerFields>

I then pull these out using rsyslog's RFC5424 parsing module, mmpstrucdata, to create json tree. The rsyslog.conf template for accessing them looks like:

template(name="jsondump" type="string" string="'%$!rfc5424-sd!mdc@18060!thread%', '%$!rfc5424-sd!mdc@18060!priority%', '%$!rfc5424-sd!mdc@18060!category%', '%$!rfc5424-sd!mdc@18060!exception%'")

I was just trying to do the same and thought I'd share what worked for me. - Sam

Answer 5

I don't believe you can use a pattern on the basic Syslog appender.

From the docs it states that

"SyslogAppender is a SocketAppender that writes its output to a remote destination specified by a host and port in a format that conforms with either the BSD Syslog format or the RFC 5424" http://logging.apache.org/log4j/2.x/manual/appenders.html#SyslogAppender

However, it does allow you to specify "format = RFC 5424"

If you use RFC 5424

Then you can put a PatterLayout in the loggerFields parameter. See http://logging.apache.org/log4j/2.x/manual/layouts.html#RFC5424Layout

Hope that helps!