JSON Variants (Log4J) with LogStash

问题:

I'm not sure if this is a follow-up or separate question to this one. There is some piece about LogStash that is not clicking. For that, I apologize for a related question. Still, I'm going out of my mind here.

I have an app that writes logs to a file. Each log entry is a JSON object. An example of my .json file looks like the following:

{
  "logger":"com.myApp.ClassName",
  "timestamp":"1456976539634",
  "level":"ERROR",
  "thread":"pool-3-thread-19",
  "message":"Danger. There was an error",
  "throwable":"java.Exception"
},
{
  "logger":"com.myApp.ClassName",
  "timestamp":"1456976539649",
  "level":"ERROR",
  "thread":"pool-3-thread-16",
  "message":"I cannot go on",
  "throwable":"java.Exception"
}

This format is what's created from Log4J2's JsonLayout. I'm trying my damnedest to get the log entries into LogStash. In an attempt to do this, I've created the following LogStash configuration file:

input {
  file {
    type => "log4j"
    path => "/logs/mylogs.log"
  }
}
output {
  file {
    path => "/logs/out.log"
  }
}

When I open /logs/out.log, I see a mess. There's JSON. However, I do not see the "level" property or "thread" property that Log4J generates. An example of a record can be seen here:

{"message":"Danger. There was an error","@version":"1","@timestamp":"2014-04-08T17:20:10.035Z","type":"log4j","host":"ip-myAddress","path":"/logs/mylogs.log"}

Sometimes I even get parse errors. I need my properties to still be properties. I do not want them crammed into the message portion or the output. I have a hunch this has something to do with Codecs. Yet, I'm not sure. I'm not sure if I should change the codec on the logstash input configuration. Or, if I should change the input on the output configuration. I would sincerely appreciate any help as I'm getting desperate at this point.

回答1:

Can you change your log format?

After I change your log format to

{ "logger":"com.myApp.ClassName", "timestamp":"1456976539634", "level":"ERROR", "thread":"pool-3-thread-19", "message":"Danger. There was an error",  "throwable":"java.Exception" }
{ "logger":"com.myApp.ClassName", "timestamp":"1456976539649", "level":"ERROR", "thread":"pool-3-thread-16", "message":"I cannot go on", "throwable":"java.Exception" }

One json log per one line and without the "," at the end of the log, I can use the configuration below to parse the json message to correspond field.

input {
    file {
       type => "log4j"
       path => "/logs/mylogs.log"
       codec => json
    }
}

回答2:

input {
    file {
        codec => json_lines { charset => "UTF-8" }
        ...
    }
}

should do the trick

回答3:

Use Logstash's log4j input.

http://logstash.net/docs/1.4.2/inputs/log4j

Should look something like this:

input {
    log4j {
        port => xxxx
    }
}

This worked for me, good luck!

回答4:

I think @Ben Lim was right, your Logstash config is fine, just need to properly format input JSON to have each log event in a single line. This is very simple with Log4J2's JsonLayout, just set eventEol=true and compact=true. (reference)