Get random records with Doctrine

Question

I wonder how to get a random number of Members from a Group, but I do not know what is the best way to do this, and I think ORDER BY RAND() is not the best alternative, as a Group can have more than 100,000 Members, performing this type of query could be very slow.

I found this way to make using SQL, but I do not know how to do the same thing in DQL: How can i optimize MySQL's ORDER BY RAND() function?

Answer 1

You could use the query you found in order to efficiently retrieve the ids of N random records via a native sql query, then do a doctrine query in order to fetch the objects via a WHERE IN(...) using dql.

Example:

// fetch $randomIds via native sql query using $em->getConnection()->... methods
// or from a memory based cache

$qb = $em->createQueryBuilder('u');

$em->createQuery('
    SELECT u
    FROM Entity\User
    WHERE ' . $qb->expr()->in('u.id', $randomIds) . '
');

The same strategy applies if you fetch random ids from a cache (like redis, maybe using SRANDMEMBER) - first fetch the ids, then fetch the entities via a WHERE IN.

You just have to make sure your cached ids are in sync with the database (deleted ids are removed from database and from cache etc.)

Answer 2

To not decrease performances I generally do as follows:

//Retrieve the EntityManager first
$em = $this->getEntityManager();

//Get the number of rows from your table
$rows = $em->createQuery('SELECT COUNT(u.id) FROM AcmeUserBundle:User u')->getSingleScalarResult();

$offset = max(0, rand(0, $rows - $amount - 1));

//Get the first $amount users starting from a random point
$query = $em->createQuery('
                SELECT DISTINCT u
                FROM AcmeUserBundle:User u')
->setMaxResults($amount)
->setFirstResult($offset);

$result = $query->getResult();  

Of course, the $amount users object you will retrieve are consecutive (i.e. the i-th, (i+1)-th,...,(i+$amount)-th), but usually there is the need of taking one or two entities at random, not the whole list. Hence, I think that this is an effective alternative.

Answer 3

I'm not aware of any way to ORDER BY RAND() "efficiently" from Doctrine. In your situation, the best thing is probably to get primary keys first, shuffle these keys and then make use of them within an IN statement.

You could also add a caching layer where to put (a subset of) the keys from the first query, especially if you have many records, so to avoid repeating the query on keys each time.