Elasticsearch: Paginating a stream of results

2019-04-30 00:34发布

问题:

My settings

Consider a stream of logs continuously sent to an Elasticsearch cluster using Filebeat.

My problem

I would like to continuously read the stream of logs from Elasticsearch. Needless to say, ES does not feature a streaming API, so I have to paginate using consecutive HTTP calls.

What have I tried

  • Using from / size
  • Using Search After

Query body for reference

{
  "size": 5,
  "from": 0,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "source:*.log",
            "analyze_wildcard": true
          }
        }
      ]
    }
  },
  "_source": {
      "include": ["message", "@timestamp"],
      "exclude": "_*"
  }
}

My question

How can I paginate a stream of log entries from Elasticsearch?