Mongo $lookup filter using nested query [duplicate

Question

This question already has an answer here:

How to use $lookup as INNER JOIN in MongoDB Aggregation? (1 answer)

Closed 9 months ago.

I'm playing with mongoDB's $lookup function, specifically the pipeline syntax, to allow me to perform some more complex queries than the ORM I am using (Sails/Waterline) allows.

A cut down version of my data looks like....

    // 'job' collection
    {
      "id"      : j1,
      "mediaID" : "ABC1234"
    },
    {
      "id"      : j2,
      "mediaID" : "DEF1234"
    },
    {
      "id"      : j3,
      "mediaID" : "FGH3456"
    }

..and..

    // 'task' collection

    // j1 tasks
    {
      "id"      : "t1",
      "job"     : "j1",
      "taskName": "MOVE",
      "status"  : "COMPLETE"
    },
    {
      "id"      : "t2",
      "job"     : "j1",
      "taskName": "PUBLISH",
      "status"  : "COMPLETE"
    },
    // j2 tasks
    {
      "id"      : "t3",
      "job"     : "j2",
      "taskName": "MOVE",
      "status"  : "FAILED"
    },
    // j3 tasks
    {
      "id"      : "t4",
      "job"     : "j3",
      "taskName": "MOVE",
      "status"  : "COMPLETE"
    }

..where the task collection links to the job collection via job.id -> task.job

What I would like to achieve is, to be able to filter jobs by job.mediaID and/or also task.status. The query I currently have almost gets what I want, but it doesn't filter out the jobs, it just doesn't populate the tasks part.

My current query is as follows...

    let results = await jobCollection.aggregate([

      // First filter jobs with matching criteria
      {
        $match: {
          $text: {$search: "1234"}
        }
      },

      // Next, link the task collection, and filter by status
      {
        "$lookup": {
          from    : 'task',
          'let'   : {job_id: '$_id'},
          pipeline: [
            {
              $match: {
                $expr: {
                  $and: [
                    // This does the linking of job.id -> task.job_id
                    {$eq: ['$job', '$$job_id']},
                    // This filters out unwanted tasks
                    {$eq: ['$status', 'FAILED']}
                  ]
                }
              }
            }
          ],
          as      : 'tasks'
        }
      }
    ])
    .toArray();

In that example, the first stage would match j1 and j2 as they both contain "1234", then I would like to further filter out the jobs, based on the task status, for example, only j2 has a task that has status==FAILED, so my end result would just be the j2 document.

I hope this makes. I think I may just need to add some clever projection on the end. Thanks

Answer 1

The $match inside $lookup pipeline has no relation with the jobCollection documents. It only filters the documents for tasks collection. So, You have to use one more $match stage after the $lookup to filter out the ROOT(jobCollection) documents.

jobCollection.aggregate([
  { "$match": { "$text": { "$search": "1234" }}},
  { "$lookup": {
    "from": "task",
    "let": { "job_id": "$_id" },
    "pipeline": [
      { "$match": {
        "$expr": {
          "$and": [
            { "$eq": ["$job", "$$job_id"] },
            { "$eq": ["$status", "FAILED"] }
          ]
        }
      }}
    ],
    "as": "tasks"
  }},
  { "$match": { "tasks": { "$ne": [] }}},
])