how to grep particulr field from logstash output

Question

I am trying to grep only few fields from this output from logstash 1.repositories#create 2.\"repo\":\"username/reponame\" . please share your ideas to grep particular info from this outpput and assign this to another variable

"message" => "<190>Nov 01 20:35:15 10-254-128-66 github_audit: {\"actor_ip\":\"192.168.1.1\",\"from\":\"repositories#create\",\"actor\":\"myuserid\",\"repo\":\"username/reponame\",\"action\":\"staff.repo_route\",\"created_at\":1516286634991,\"repo_id\":44743,\"actor_id\":1033,\"data\":{\"actor_location\":{\"location\":{\"lat\":null,\"lon\":null}}}}",

I am using this syslog.conf file to get the output.

input {
  tcp {
    port => 8088
    type => syslog
  }
  udp {
    port => 8088
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}"
    }
    grep {
      match => { "message" => "repositories#create" }
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

I am not able to add my comments for your reply, thank you so much for your reply.

could you please share your ideas to get username: and repo: only from this output , i m trying assign the values from this particular output, thanks again

message: "github_audit: {"actor_ip":"192.168.1.1","from":"repositories#create","actor":"username","repo":"username/logstashrepo","user":"username","created_at":1416299104782,"action":"repo.create","user_id":1033,"repo_id":44744,"actor_id":1033,"data":{"actor_location":{"location":{"lat":null,"lon":null}}}}", @version: "1", @timestamp: "2014-11-18T08:25:05.427Z", host: "15-274-145-63", type: "syslog", syslog5424_pri: "190", timestamp: "Nov 18 00:25:05", actor_ip: "10.239.37.185", from: "repositories#create", actor: "username", repo: "username/logstashrepo", user: "username", created_at: 1416299104782, action: "repo.create", user_id: 1033, repo_id: 44744, actor_id: 1033,

Answer 1

Use a grok filter to extract the JSON payload into a separate field, then use a json filter to extract the fields from the JSON object. The example below works but only extracts the JSON payload from messages prefixed with "github_audit: ". I'm also guessing that the field after the timestamp is a hostname that should overwrite whatever might currently be in the "host" field. Don't forget to add a date filter to parse the string in the "timestamp" field into "@timestamp".

filter {
  grok {
    match => [
      "message",
      "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:host} %{GREEDYDATA:message}"
    ]
    overwrite => ["host", "message"]
  }
  if [message] =~ /^github_audit: / {
    grok {
      match => ["message", "^github_audit: %{GREEDYDATA:json_payload}"]
    }
    json {
      source => "json_payload"
      remove_field => "json_payload"
    }
  }
}