My settings
Consider a stream of logs continuously sent to an Elasticsearch cluster using Filebeat.
My problem
I would like to continuously read the stream of logs from Elasticsearch. Needless to say, ES does not feature a streaming API, so I have to paginate using consecutive HTTP calls.
What have I tried
- Using
from
/size
- Using Search After
Query body for reference
{
"size": 5,
"from": 0,
"sort": [
{
"@timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "source:*.log",
"analyze_wildcard": true
}
}
]
}
},
"_source": {
"include": ["message", "@timestamp"],
"exclude": "_*"
}
}
My question
How can I paginate a stream of log entries from Elasticsearch?