Mongoose insertMany limit

Question

Mongoose 4.4 now has an insertMany function which lets you validate an array of documents and insert them if valid all with one operation, rather than one for each document:

var arr = [{ name: 'Star Wars' }, { name: 'The Empire Strikes Back' }]; Movies.insertMany(arr, function(error, docs) {});

If I have a very large array, should I batch these? Or is there no limit on the size or array?

For example, I want to create a new document for every Movie, and I have 10,000 movies.

Answer 1

Mongo 3.6 update:

The limit for insertMany() has increased in Mongo 3.6 from 1000 to 100,000. Meaning that now, groups above 100,000 operations will be divided into smaller groups accordingly. For example: a queue that has 200,000 operations will be split and Mongo will create 2 groups of 100,000 each.

Answer 2

I'd recommend based on personal experience, batch of 100-200 gives you good blend of performance without putting strain on your system.

insertMany group of operations can have at most 1000 operations. If a group exceeds this limit, MongoDB will divide the group into smaller groups of 1000 or less. For example, if the queue consists of 2000 operations, MongoDB creates 2 groups, each with 1000 operations.

The sizes and grouping mechanics are internal performance details and are subject to change in future versions.

Executing an ordered list of operations on a sharded collection will generally be slower than executing an unordered list since with an ordered list, each operation must wait for the previous operation to finish.

Answer 3

The method takes that array and starts inserting them through the insertMany method in MongoDB , so the size of the array itself actually depends on how much your machine can handle .

But please note that there is another point, which is not a limitation but something worth keeping into consideration, on how MongoDB deals with multiple operations, by default it handles a batch of 1000 operations at a time and splits whats more than that.