ElasticSearch query using NEST 2.x with scroll is

Question

I'm trying to retrieve all data from the elasticsearch based on a message occurrence, i figured that if i used Scroll i could loop until the document search end but the following query returns Documents = 0 but Total = 1954:

var response = client.Search<Log4Net>(s => s
                                            .Query(q => q.QueryString(qs => qs
                                             .DefaultField(m => m.Message).Query("\"" + message + "\"")))
                                             .SearchType(SearchType.Scan)
                                             .Scroll("60s"));
        while (response.Documents.Any())
        {
            var request = new BulkRequest();
            request.Refresh = true;
            request.Consistency = Consistency.One;
            request.Operations = new List<IBulkOperation>();
            foreach (var item in response.Documents)
            {
                request.Operations.Add(new BulkIndexOperation<Log4Net>(item));
            }

            var result = client.Bulk(request);

            response = client.Scroll<Log4Net>("60s", response.ScrollId);
        }

the response.Document is coming empty if i use the scroll, if i remove and get the first 1000 messages i can get the data, is anything wrong with how i'm using the Scroll?

Answer 1

If you specify .SearchType(SearchType.Scan), the first response doesn't contain any documents; It will give you the total documents in the .Total property that will be returned by scrolling using the .ScrollId on the response in a scroll request.

If you don't specify .SearchType(SearchType.Scan), the first response will contain the first set of documents.

This is a difference in Elasticsearch and not NEST. SearchType.Scan is actually deprecated in 2.1.0, but is still in NEST 2.x as it supports all minor versions of Elasticsearch 2.x.