Add fields to Logstash Twitter input and Elasticse

Question

I am using logstash to save the twitter stream to elasticsearch. Before saving, I want to

1. Add a new field which indicates whether the tweet is a RT or reply or organic
2. Use the tweet id as elasticsearch's document id

But I've been unable to do either! Logstash config file:

input {
twitter {
    oauth_token => ""
    oauth_token_secret => ""
    consumer_key => ""
    consumer_secret => ""
    full_tweet => true
    keywords => ["test"]
}
}

filter {
ruby {
    code => "
        if !event['retweeted_status'].nil?
            event['tweet_type'] = 'Retweet'
        elsif !event['in_reply_to_screen_name'].nil?
            event['tweet_type'] = 'Reply'
        else
            event['tweet_type'] = 'Organic'
        end
    "
}
}

output {
elasticsearch {
    document_id => [id]
    index_type => "twitter"
    protocol => "http"
    bind_host => "127.0.0.1"
}
}

What am I doing wrong?

Answer 1

You don't need to drop to ruby to test fields. Try:

if [retweeted_status] {
    mutate {
       add_field => { "tweet_type", "Retweet" }
    }
}

(NOTE: that's pseudo-code; I may have the the {s and => wrong).

As for using the document id, try:

document_id => "%{id}"

share|improve this answer

answered Mar 7 '15 at 21:09

Alain CollinsAlain Collins

14.1k12349