Extracting fields from paths in logstash

Question

I am configuring logstash to collect logs from multiple workers on multiple hosts. I'm currently adding fields for host:

input {
    file {
            path => "/data/logs/box-1/worker-*.log"
            add_field => {
                "original_host" => "box-1" 
            }
    }
    file {
            path => "/data/logs/box-2/worker-*.log"
            add_field => {
                "original_host" => "box-2"
            }
    }

However, I'd also like to add a field {'worker': 'A'} and so on. I have lots of workers, so I don't want to write a file { ... } block for every combination of host and worker.

Do I have any alternatives?

Answer 1

You should be able to do a path => "/data/logs/*/worker-*.log" and then add a grok filter to pull out what you need.

filter { grok { match => [ "path", "/(?<original_host>[^/]+)/worker-(?<worker>.*).log" ] } }

or something very close to that.... might want to surround it with if [path] =~ /worker/ depending on what else you have in your config file.