Cache/Re-Use a Subquery in MySQL

Question

I have a very complex MySQL query that includes use of the same subquery three times. Will MySQL actually run the subquery three times? (It's an expensive one.) If so, is there a way for me to tell MySQL to save or cache the results so it won't do that? I could save the data in a large array then re-feed it to MySQL, but I'd rather not move it out and back into the database like that.

This is the subquery that appears three times:

SELECT id FROM programs 
WHERE submitter_id=32 AND id in (
    SELECT id FROM programs 
    WHERE feed_id=2478 AND id in (
        SELECT program_id FROM playlist_program_map 
        WHERE playlist_id=181)))

And here's an example of the full query in which the query appears:

SELECT object_id, programs.created AS created, 
MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE) AS relevance 
FROM comments_programs USE INDEX (text) 
LEFT JOIN programs ON programs.id=object_id 
WHERE object_id IN (
    SELECT id FROM programs 
    WHERE 1 AND id IN (
        SELECT id FROM programs 
        WHERE submitter_id=32 AND id in (
            SELECT id FROM programs 
            WHERE feed_id=2478 AND id in (
                SELECT program_id FROM playlist_program_map 
                WHERE playlist_id=181)))) 
AND MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE)>0)

UNION (

SELECT object_id, programs.created AS created, 
MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE) AS relevance 
FROM descriptions_programs USE INDEX (text) 
LEFT JOIN programs ON programs.id=object_id 
WHERE object_id IN (
    SELECT id FROM programs 
    WHERE 1 AND id IN (
        SELECT id FROM programs 
        WHERE submitter_id=32 AND id in (
            SELECT id FROM programs 
            WHERE feed_id=2478 AND id in (
                SELECT program_id FROM playlist_program_map 
                WHERE playlist_id=181)))) 
AND MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE)>0 AND current=1 ) 

UNION (

SELECT object_id, programs.created AS created, 
MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE) AS relevance 
FROM titles_programs USE INDEX (text) 
LEFT JOIN programs ON programs.id=object_id 
WHERE object_id IN (
    SELECT id FROM programs 
    WHERE 1 AND id IN (
        SELECT id FROM programs 
        WHERE submitter_id=32 AND id in (
            SELECT id FROM programs 
            WHERE feed_id=2478 AND id in (
                SELECT program_id FROM playlist_program_map 
                WHERE playlist_id=181)))) 
AND MATCH(text) AGAINST ('excellent ' IN BOOLEAN MODE)>0 AND current=1;

Answer 1

For whatever reason, mysql IN clauses with a sub-select run very slow. Better to use join. Your sub-query becomes:

SELECT id from programs P1 INNER JOIN programs P2 ON P1.id = P2.id INNER JOIN playlist_program_map PMAP ON P2.id = PMAP.program_id WHERE P1.submitter_id=32 AND P2.feed_id=2478 AND PMAP.playlist_id=181

It will run much faster.

Answer 2

See what EXPLAIN EXTENDED says.

If it says DEPENDENT SUBQUERY or UNCACHEABLE SUBQUERY, then it will be reevaluated each time it's used.

This happens if the subquery uses session variables or is a correlated subquery.

If it doesn't, it most probably will be cached.

If your case the subquery will not be cached, it will be reevaluated in each UNION'ed set.

You subquery, though, seems to be too complicated. Why don't you just use:

SELECT id
FROM   playlist_program_map ppm, programs p
WHERE  ppm.playlist_id = 181
       AND p.id = ppm.program_id
       AND submitter_id = 32
       AND feed_id = 2478

If you have an index on playlist_program_map (playlist_id), this query should work like a charm.

Could you please tell me two more things:

1. How many rows are there in playlist_program_map and how many DISTINCT playlist_id values are there?
   • How many rows are there in programs and how many DISTINCT submitter_id, feed_id pairs are there?

From your comment I can conclude that there are 10 programs per playlist in average, and 200 programs per (submitter, feed) pair. This means your index on playlist_program_map is more selective than the one on (submitter, feed), and playlist_program_map must be leading in the join.

The fulltext index in your case also doesn't seem to be very selective, given that you need to join 10 programs out of 2,000,000.

You may better try the following:

SELECT object_id, programs.created AS created
FROM   playlist_program_map ppm, programs p, comments_programs cp
WHERE  ppm.playlist_id = 181
       AND p.id = ppm.program_id
       AND p.submitter_id = 32
       AND p.feed_id = 2478
       AND cp.object_id = p.id
       AND cp.text REGEXP 'excellent'

, and repeat this for all three tables.