Telegraf tail with grok pattern error

Question

I am using Telegraf to get logs information from Apache NiFi, for this task I am using this config:

[[inputs.tail]]
  ## files to tail.
  files = ["/var/log/nifi/nifi-app.log"]
  ## Read file from beginning.
  from_beginning = true
  #name_override = "nifi_app"

  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "grok"
  grok_patterns = [ "%{DATE:date} %{TIME:time} %{WORD:EventType} \[%{GREEDYDATA:NifiTask} %{NOTSPACE:Thread}\] %{NOTSPACE:NifiEventType} %{GREEDYDATA:EventText} %{NUMBER:EventDuration} %{WORD:EventDurationUnits}" ]

When I try to start telegraf it give me this error:

Error parsing /etc/telegraf/telegraf.conf, toml: line 10: parse error

The pattern I wrote was tested in a Grok debugger with this text:

2018-08-02 10:53:16,976 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 11863 nanos

These are the results of some testing:

grok_patterns = ["\[%{GREEDYDATA:NifiTask}\]"] ==> toml: line 10: parse error  
grok_patterns = ["[%{GREEDYDATA:NifiTask}]"] ==> Invalid data format: grok  
grok_patterns = ['\[%{GREEDYDATA:NifiTask}\]'] ==> Invalid data format: grok  
grok_patterns = ["\\[%{GREEDYDATA:NifiTask}\\]"] ==> Invalid data format: grok  
grok_patterns = ['[%{GREEDYDATA:NifiTask}]'] -> Invalid data format: grok  

The first option for me is the right one, but doesn't works, and the problem seems to be the way the bracket is being escaped.

How is possible to solve this issue?

Answer 1

There was more than one problem:

• First problem: the grok dataformat is added to Telegraf in the 1.8 release (ref), so I must use a nightly install until this version is released.

• Second problem: how to escape the brackets, there are problems doing it in a regular way, so what I finally did was to put this part in a custom pattern file, this way it works perfectly.