Spring data elasticsearch scan and scroll paged re

I have problems using scan and scroll as the pages from scan return strange result. I am not using them exactly as stated in the documentation. In the method getAllExampleItems() I have a while loop on the "page.hasNext()" method, which always return false. The reason for that is the Pageable from the search query is not set on the result page, and the number of pages therefore is always 1. Another thing that is weird is that I set 100 as the result size and get 500 back! Isn't that a bug or what do I do wrong? This is an example service:

package service;

import config.ElasticSearchConfig;
import items.ExampleItem;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.elasticsearch.core.ElasticsearchTemplate;
import org.springframework.data.elasticsearch.core.query.IndexQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import static org.elasticsearch.index.query.FilterBuilders.typeFilter;
import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;

@Service
public class ExampleService {

@Autowired
private ElasticsearchTemplate searchTemplate;


public List<ExampleItem> getAllExampleItems() {
    List<ExampleItem> allItems = new ArrayList<>();
    String scrollId = searchTemplate.scan(exampleItemSearchQuery(), 1000, false);
    Page<ExampleItem> page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class);
    if (page != null && page.hasContent()) {
        allItems.addAll(page.getContent());
        while (page != null && page.hasNext()) {
            page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class);
            if (page != null && page.hasContent()) {
                allItems.addAll(page.getContent());
            }
        }
    }
    return allItems;
}


private SearchQuery exampleItemSearchQuery() {
    return new NativeSearchQueryBuilder()
            .withQuery(matchAllQuery())
            .withFilter(typeFilter("exampleitem"))
            .withPageable(new PageRequest(0, 100))
            .build();
}


public void saveAllExampleItems(List<ExampleItem> items) {
         List<IndexQuery> indexQueries = new ArrayList<>();
         for (ExampleItem item : items) {
             IndexQuery qry = new IndexQuery();
             qry.setId(item.getId());
             qry.setObject(item);
             qry.setIndexName(ElasticSearchConfig.ITEMS);
             indexQueries.add(qry);
         }
         searchTemplate.bulkIndex(indexQueries);
         searchTemplate.refresh(ExampleItem.class, true);
     }
}

Could somebody explain to me why this does not work, or what I am doing wrong?

回答1:

Working example of scan and scroll with Spring data es can be found at TemplateTest Page

Reason you are getting 500 instead of 100 is because searchHit will be always

size * number_of_primary_shard in elasticsearch.

ref : es site

回答2:

I faced the same problem. Page must be not null and hasNext() is always true. hasContent() can assert the page. Replace hasNext() with hasContent() and try again....

回答3:

I faced the same issue, then I figured that the difference between my code, and the "working example", is that I used page.hasNext(), just like you did above. Remove that and try again.