logstash过滤

Question

2020-09-23 04:00:00 Local7.Info 10.0.1.22 Sep 23 04:00:19 src@Master-AD-9000-H-wit : [2020-09-23_04-00-19] NAT_LOG_DEL_ENTRY [udp]10.99.31.22:7335(113.57.168.162:7335) -> 39.65.223.170:41267(39.65.223.170:41267)

上面一行数据是我的log数据，我想把这条数据保存到es中，只保留指定数据，且数据保存到es中需要mapping

请问这种情况下 logstash怎么写配置文件

Answer 1

这是grok自带的正则地址：https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

写了个示例。正则挺耗资源的，注意你的写入速率哈。

input {
    file {
        path => ["/opt/test/test.log"]
        start_position => beginning
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => true
            what => previous
         }
    }
}
filter{
  grok{
    match => {"message" => ".*?\[%{YEAR:year}-%{MONTHNUM2:month}-%{MONTHDAY:day}_(?<time>(?:%{HOUR}-%{MINUTE}-%{SECOND}))\].*?%{IP:innerIp}:%{POSINT:innerPort}\(%{IP:natIp}:%{POSINT:natPort}\).*?%{IP:outIp}:%{POSINT:outPort}\(%{IP:outIp2}:%{POSINT:outPort2}\)"}
  }
  mutate {
    remove_field => ["message"]
    remove_field => ["path"]
    remove_field => ["host"]
  }
}
output {
  stdout {}
 # elasticsearch {
 #   hosts => ["http://192.168.0.153:9200"]
 #   index => "_test-%{+YYYY.MM.dd}"
 # }
}