mongo aggregate result exceed maximum document siz

Question

I use mongo aggregate function to find duplicated documents in a collection, where the collections looks like the following:

{_id, placement_id, placement_name, program_id, target}

I need to find all the documents that have exactly the same fields except _id and placement_id, so this two documents are the same:

{_id:3, placement_id:23, placement_name:"pl1", program_id:5, target:"-"}
{_id:7, placement_id:55, placement_name:"pl1", program_id:5, target:"-"}

The aggregate function I came up with is:

db.placements.aggregate({$group:{_id:{placement_name:"$placement_name", program_id:"$program_id", target:"$target"}, total:{$sum:1}}},{$match:{total:{$gte:2}}});

Then mongo just returned:

Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at DBCollection.aggregate (src/mongo/shell/collection.js:897:9)
    at (shell):1:15
Wed Apr  2 07:43:23.090 aggregate failed: {
    "errmsg" : "exception: aggregation result exceeds maximum document size (16MB)",
    "code" : 16389,
    "ok" : 0
} at src/mongo/shell/collection.js:898

the aggregate is correct, I tested it on a smaller collection and it works fine, but the production collection has about 80M documents. I was wondering when trying the find() function on 80M documents, it works and asking you to type 'it' for more records. How come the aggregate function doesn't have this capability? I also tried to append limit() to the end of the aggregate function but it won't work either. Any work around? Thanks.

Answer 1

how come aggregate function don't have this capability?

It does in 2.6 which will be out very soon.

i also tried to append limit() to the end of the aggregate function, it won't work either. any work around?

There is a $limit operator for the aggregation framework: http://docs.mongodb.org/manual/reference/operator/aggregation/limit/

Answer 2

I was running into the same issue there. After googling I found out that you can use the runCommand in Mongo to use the aggregation and set the allowDiskUse option to true, to use the hard drive as an extension of the memory. enter link description here