Random order & pagination Elasticsearch

Question

In this issue is a feature request for ordering with optional seed allowing for recreation of random order.

I need to be able to paginate random ordered results. How could this be be done with Elasticsearch 0.19.1 ?

Thanks.

Answer 1

Good solution from imotov.

Here is something much more simple and you don't need to rely in a document property:

{
  "query" : { "query_string" : {"query" : "*:*"} },
  "sort" : {
    "_script" : { 
        "script" : "Math.random()",
        "type" : "number",
        "params" : {},
        "order" : "asc"
    }
  }
}

if you want to set a range that would be something like:

{
  "query" : { "query_string" : {"query" : "*:*"} },
  "sort" : {
    "_script" : { 
        "script" : "Math.random() * (myMax - myMin) + myMin",
        "type" : "number",
        "params" : {},
        "order" : "asc"
    }
  }
}

replacing the max and min with your proper values.

Answer 2

I ended up solving it slightly different than what imotov suggested. As I have multiple clients I didn't want to implement the logic surrounding the salt string on every one of them.

I already had a randomized_key on the model. I also didn't need the order to be random for every request so I made a scheduled job to update the randomized key every night and then sorted by that field in Elasticssearch.

Answer 3

You can sort using a hash function of a unique field (for example id) and a random salt. Depending on how truly random the results should be, you can do something as primitive as:

{
  "query" : { "query_string" : {"query" : "*:*"} },
  "sort" : {
    "_script" : { 
        "script" : "(doc['_id'].value + salt).hashCode()",
        "type" : "number",
        "params" : {
            "salt" : "some_random_string"
        },
        "order" : "asc"
    }
  }
}

or something as sophisticated as

{
  "query" : { "query_string" : {"query" : "*:*"} },
  "sort" : {
    "_script" : { 
        "script" : "org.elasticsearch.common.Digest.md5Hex(doc['_id'].value + salt)",
        "type" : "string",
        "params" : {
            "salt" : "some_random_string"
        },
        "order" : "asc"
    }
  }
}

The second example will produce more random results but will be somewhat slower.

For this approach to work the field _id has to be stored. Otherwise, the query will fail with NullPointerException.

Answer 4

This should be considerably faster than both answers above and supports seeding:

curl -XGET 'localhost:9200/_search' -d '{
  "query": {
    "function_score" : {
      "query" : { "match_all": {} },
      "random_score" : {}
    }
  }
}';

See: https://github.com/elasticsearch/elasticsearch/issues/1170

Answer 5

Well, i was looking at doing this and all the approaches above seemed a little "too complicated" for something that should be relatively simple. So i came up with an alternative that works perfectly well without the need of "going mental"

I perform a _count query first then combine it with "Start" and rand(0,$count)

e.g.

JSONArray = array of json to send to ElasticSearch
$total_results = $ElasticSearchClient->count(JSONArray)
$start = rand(0, $total_results)
JSONArray['body']['from'] = $start;
$ElasticSearchClient->search(JSONArray);

Assumptions for the above example:

• You're running PHP
• You're also using the the PHP Client

But you dont NEED to do this with PHP, the approach would work with any example.