Logstash: how to use filter to match filename when

Question

I am new to logstash. I have some logs stored in AWS S3 and I am able to import them to logstash. My question is: is it possible to use the grok filter to add tags based on the filenames? I try to use:

grok { 
    match => {"path" => "%{GREEDYDATA}/%{GREEDYDATA:bitcoin}.err.log"}
    add_tag => ["bitcoin_err"]
    }

This is not working. I guess the reason is "path" only working with file inputs.

Here is the structure of my S3 buckets:

my_buckets
    ----A
        ----2014-07-02
            ----a.log
            ----b.log
    ----B
        ----2014-07-02
            ----a.log
            ----b.log

I am using this inputs conf:

s3 {
    bucket => "my_buckets"
    region => "us-west-1"
    credentials => ["XXXXXX","XXXXXXX"]
    }

What I want is that, for any log messages in:

• "A/2014-07-02/a.log": they will have tag ["A","a"].

• "A/2014-07-02/b.log": they will have tag ["A","b"].

• "B/2014-07-02/a.log": they will have tag ["B","a"].

• "B/2014-07-02/b.log": they will have tag ["B","b"].

Sorry about my english....

Answer 1

There is no "path" in S3 inputs. I mount the S3 storage on my server and use the file inputs. With file inputs, I can use the filter to match the path now.

Answer 2

With Logstash 6.0.1, I was able to get key for each file from S3. In your case, you can use this key (or path) in filter to add tags.

Example:

input {
    s3 {
        bucket => "<bucket-name>"
        prefix => "<prefix>"
    }
}

filter {
    mutate {
        add_field => {
            "file" => "%{[@metadata][s3][key]}"
        }
    }
    ...
}

Use this above file field in filter to add tags.

Reference:

Look for eye8 answer in this issue

Answer 3

I updated the s3 input plugin.

Check this link

I updated "process_local_log" method.

Answer 4

If you want to use tags based on filename, I think that this will work (I have not test it):

filter {
  grok {
    match => [ "path", "%{GREEDYDATA:content}"]   
  }     
  mutate {
    add_tag => ["content"]
  }
}

"content" tag will be the filename, now it's up to you to modify the pattern to create differents tags with the specific part of the filename.