Get a count of total documents with MongoDB when u

Question

I am interested in optimizing a "pagination" solution I'm working on with MongoDB. My problem is straight forward. I usually limit the number of documents returned using the limit() functionality. This forces me to issue a redundant query without the limit() function in order for me to also capture the total number of documents in the query so I can pass to that to the client letting them know they'll have to issue an additional request(s) to retrieve the rest of the documents.

Is there a way to condense this into 1 query? Get the total number of documents but at the same time only retrieve a subset using limit()? Is there a different way to think about this problem than I am approaching it?

Answer 1

Times have changed, and I believe you can achieve what the OP is asking by using aggregation with $sort, $group and $project. For my system, I needed to also grab some user info from my users collection. Hopefully this can answer any questions around that as well. Below is an aggregation pipe. The last three objects (sort, group and project) are what handle getting the total count, then providing pagination capabilities.

db.posts.aggregate([
  { $match: { public: true },
  { $lookup: {
    from: 'users',
    localField: 'userId',
    foreignField: 'userId',
    as: 'userInfo'
  } },
  { $project: {
    postId: 1,
    title: 1,
    description: 1
    updated: 1,
    userInfo: {
      $let: {
        vars: {
          firstUser: {
            $arrayElemAt: ['$userInfo', 0]
          }
        },
        in: {
          username: '$$firstUser.username'
        }
      }
    }
  } },
  { $sort: { updated: -1 } },
  { $group: {
    _id: null,
    postCount: { $sum: 1 },
    posts: {
      $push: '$$ROOT'
    }
  } },
  { $project: {
    _id: 0,
    postCount: 1,
    posts: {
      $slice: [
        '$posts',
        currentPage ? (currentPage - 1) * RESULTS_PER_PAGE : 0,
        RESULTS_PER_PAGE
      ]
    }
  } }
])

Answer 2

Try as bellow:

cursor.count(false, function(err, total){ console.log("total", total) })

core.db.users.find(query, {}, {skip:0, limit:1}, function(err, cursor){
    if(err)
        return callback(err);

    cursor.toArray(function(err, items){
        if(err)
            return callback(err);

        cursor.count(false, function(err, total){
            if(err)
                return callback(err);

            console.log("cursor", total)

            callback(null, {items: items, total:total})
        })
    })
 })

Answer 3

there is a way in Mongodb 3.4: $facet

you can do

db.collection.aggregate([
  {
    $facet: {
      data: [{ $match: {} }],
      total: { $count: 'total' }
    }
  }
])

then you will be able to run two aggregate at the same time

Answer 4

MongoDB allows you to use cursor.count() even when you pass limit() or skip().

Lets say you have a db.collection with 10 items.

You can do:

async function getQuery() {
  let query = await db.collection.find({}).skip(5).limit(5); // returns last 5 items in db
  let countTotal = await query.count() // returns 10-- will not take `skip` or `limit` into consideration
  let countWithConstraints = await query.count(true) // returns 5 -- will take into consideration `skip` and `limit`
  return { query, countTotal } 
}

Answer 5

It all depends on the pagination experience you need as to whether or not you need to do two queries.

Do you need to list every single page or even a range of pages? Does anyone even go to page 1051 - conceptually what does that actually mean?

Theres been lots of UX on patterns of pagination - Avoid the pains of pagination covers various types of pagination and their scenarios and many don't need a count query to know if theres a next page. For example if you display 10 items on a page and you limit to 13 - you'll know if theres another page..

Answer 6

It is possible to get the total result size without the effect of limit() using count() as answered here: Limiting results in MongoDB but still getting the full count?

According to the documentation you can even control whether limit/pagination is taken into account when calling count(): https://docs.mongodb.com/manual/reference/method/cursor.count/#cursor.count

Edit: in contrast to what is written elsewhere - the docs clearly state that "The operation does not perform the query but instead counts the results that would be returned by the query". Which - from my understanding - means that only one query is executed.

Example:

> db.createCollection("test")
{ "ok" : 1 }

> db.test.insert([{name: "first"}, {name: "second"}, {name: "third"}, 
{name: "forth"}, {name: "fifth"}])
BulkWriteResult({
    "writeErrors" : [ ],
    "writeConcernErrors" : [ ],
    "nInserted" : 5,
    "nUpserted" : 0,
    "nMatched" : 0,
    "nModified" : 0,
    "nRemoved" : 0,
    "upserted" : [ ]
})

> db.test.find()
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c8"), "name" : "forth" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c9"), "name" : "fifth" }

> db.test.count()
5

> var result = db.test.find().limit(3)
> result
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }

> result.count()
5 (total result size of the query without limit)

> result.count(1)
3 (result size with limit(3) taken into account)