Parsing a date field in logstash to elastic search

Question

I am trying to parse log files from IIS to the ELK stack (Logstash:2.3, Elastic:2.3 and Kibana:4.5, CentOS 7 vm).

I have attempted to parse a date field from the log message as the event timestamp using the date filter below in my logstash configuration:

date {
    match => ["date_timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "Europe/London"
    locale => "en"
    target => "@timestamp"
} 

The first few characters of the entire log message that was parsed to Elastic Search is:

"message": "2016-03-01 03:30:49  .........

The date field above was parsed to Elastic Search as:

"date_timestamp": "16-03-01 03:30:49",

However, the event timestamp that was parsed to Elastic Search using the date filter above is:

"@timestamp": "0016-03-01T03:32:04.000Z",

I will like the @timestamp to be exactly 2016-03-01T03:30:49 as I can't immediately figure out why there is a difference between the hours and minutes.

I have looked at similar problems and documentations such as this one on SO and this one on logstash documentation and logstash documentation.

Any pointer in the right direction will be appreciated.

Regards

SO

Answer 1

in your date_timestamp you have only 2 characters for year: "16-03-01 03:30:49", so the date pattern in your date filter is incorrect, should be:

date {
    match => ["date_timestamp", "yy-MM-dd HH:mm:ss"]
    timezone => "Europe/London"
    locale => "en"
    target => "@timestamp"
}