I'm trying to copy a small subset of data from index A into index B.
I have a Logstash config running on an EC2 instance with the Elasticsearch input, some filters and the Elasticsearch output.
This works fine the first time I run it, but if any new data is added to Index A (that meets the input's query criteria) the Logstash config doesn't detect it and output it to Index B.
This has me thinking, does the Elasticsearch input only run once? Unlike inputs like the file or S3 input that will pick up new files added.
The Logstash elasticsearch input plugin does a single search (via scan/scroll) with an optional query and when done it quits.
You're right in that it doesn't behave the same way as the for file input plugin, which will simply tail one or more files and handle the new lines on the fly.
You might want to investigate the Watcher plugin if you need a way to be notified when a new document matches a query. Watcher could well detect the presence of a new document matching a query and then call a webhook, which could be materialized by the http input plugin in your Logstash.
