Different structure in a few lines in my log file

Question

My log file contains different structures in a few lines, and I can not grok it, I don't know if we can test by lines or attribute, I'm still a beginner. if you don't understand me I can give you some examples :

input :

id=firewall action=bloc type=web
id=firewall fw="ER" type=filter
id=firewall fw="Az" tz="loo" action=bloc

Pattern:

id=%{WORD:id} ...

I thought to add some patterns between ()?, but i don't know exactly how to do it. you can use this site to test it http://grokdebug.herokuapp.com/

Any help please? What should i do :(

Answer 1

Not sure if I understood well your question but I will try to answer. I think the first thing you have to do is to parse the different fields from your input. Example of pattern to parse your first line input :

PATTERN %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} (in $LOGSTASH_HOME/pattern/extra)

Then in your logstash configuration file :

filter {
    grok {
        patterns_dir => "$LOGSTASH_HOME/pattern"
        match => [ "message" => "%{PATTERN}" ]
    }
}

This will match your first line as 3 fields ("id=firewall" "action=bloc" "type=web") (you have to adapt it if you have more than 3 fields).

And the last thing you seem be looking for is splitting field (in key-value scheme) like id=firewall would become id => "firewall". This can be done with the kv plugin. I never used it but I recommend you the logstash docs here

If I did not understand you question, please be more clear.

Answer 2

Logstash supports key-value Values, take a look at http://logstash.net/docs/1.4.2/filters/kv.

Or you could use multiple match values:

grok {
    patterns_dir => "./patterns"
    match => [
        "message", "%{BASE_PATTERN} %{EXTRA_PATTERN}",
        "message", "%{BASE_PATTERN}",
        "message", "%{SOME_OTHER_PATTERN}"
    ]
}