Logstash Elasticsearch Input Plugin for streaming

Question

I would like to know if we can use logstash-input-elasticsearch plugin for streaming data , ex: if I have data available in my database and I run ElasticSearch input plugin , it will index the data into an output, but if after some time more data comes of ElasticSearch database , Is ElasticSearch input plugin is able to index that data without restarting the logstash ?

Thank you for your attention and your help.

Answer 1

By default, the elasticsearch input will run a scroll query on your ES instance, pump the documents matching the query, stream them to your outputs and when done Logstash will shut down.

However, what you can do is set up a cron job that will start your Logstash pipeline at regular intervals and queries the new documents present in ES.