How to remove trailing newline from message field

2020-04-14 07:00发布

I am shipping Glassfish 4 logfiles with Logstash to an ElasticSearch sink. How can I remove with Logstash the trailing newline from a message field?

My event looks like this:

{
  "@timestamp" => "2013-11-21T13:29:33.081Z",
  "message" => "[2013-11-21T13:29:32.577+0000] [glassfish 4.0] [INFO] [] [javax.resourceadapter.mqjmsra.lifecycle] [tid: _ThreadID=142 _ThreadName=Thread-43] [timeMillis: 1385040572577] [levelValue: 800] [[\n  MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter stopped.]]\n",
  "@version" => "1",
  "tags" => ["multiline", "date_filtered"],
  "host" => "myhost",
  "path" => "../server.log"
} 

标签: logstash
2条回答
虎瘦雄心在
2楼-- · 2020-04-14 07:41

A second solution is using the mutate filter of Logstash. It allows you to strip the value of a field.

filter {
  # Remove leading and trailing whitspaces (including newline etc. etc.)
  mutate {
    strip => "message"
  }
}
查看更多
成全新的幸福
3楼-- · 2020-04-14 07:49

You have to use the multiline filter with the correct pattern, to tell logstash, that every line with precending whitespace belongs to the line before. Add this lines to your conf file.

filter{
  ...
  multiline {
    type => "gflogs"
    pattern => "\[\#\|\d{4}"
    negate => true
    what => "previous"
  }
  ...
}

You can also include grok plugin to handle timestamp and filter irregular lines from beeing indexed.

See complete stack with single logstash instance on same machine

input {
  stdin {
    type => "stdin-type"
  }

  file {
    path => "/path/to/glassfish/logs/*.log"
    type => "gflogs"
  }
}

filter{
  multiline {
    type => "gflogs"
    pattern => "\[\#\|\d{4}"
    negate => true
    what => "previous"
  }

  grok {
    type => "gflogs"
    pattern => "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:loglevel}\|%{DATA:server_version}\|%{JAVACLASS:category}\|%{DATA:kv}\|%{DATA:message}\|\#\]"
    named_captures_only => true
    singles => true
  }

  date {
    type => "gflogs"
    match => [ "timestamp", "ISO8601" ]
  }

  kv {
    type => "gflogs"
    exclude_tags => "_grokparsefailure"
    source => "kv"
    field_split => ";"
    value_split => "="
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch { embedded => true }
}

This worked for me. Pleas look also this post on logstash-usergroup. I can also advice the great and up to date logstash book. Its also a good way to support the work of the logstash author.

Hope to see you on any JUG-Berlin Event!

查看更多
登录 后发表回答