Mongoose (mongodb) batch insert?

问题:

Does Mongoose v3.6+ support batch inserts now? I've searched for a few minutes but anything matching this query is a couple of years old and the answer was an unequivocal no.

Edit:

For future reference, the answer is to use Model.create(). create() accepts an array as its first argument, so you can pass your documents to be inserted as an array.

See Model.create() documentation

回答1:

Model.create() vs Model.collection.insert(): a faster approach

Model.create() is a bad way to do inserts if you are dealing with a very large bulk. It will be very slow. In that case you should use Model.collection.insert, which performs much better. Depending on the size of the bulk, Model.create() will even crash! Tried with a million documents, no luck. Using Model.collection.insert it took just a few seconds.

Model.collection.insert(docs, options, callback)

• docs is the array of documents to be inserted;
• options is an optional configuration object - see the docs
• callback(err, docs) will be called after all documents get saved or an error occurs. On success, docs is the array of persisted documents.

As Mongoose's author points out here, this method will bypass any validation procedures and access the Mongo driver directly. It's a trade-off you have to make since you're handling a large amount of data, otherwise you wouldn't be able to insert it to your database at all (remember we're talking hundreds of thousands of documents here).

A simple example

var Potato = mongoose.model('Potato', PotatoSchema);

var potatoBag = [/* a humongous amount of potato objects */];

Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Reference

• Mongo documentation
• Aaron Heckman on Google Groups discussing bulk inserts

回答2:

Mongoose 4.4.0 now supports bulk insert

Mongoose 4.4.0 introduces --true-- bulk insert with the model method .insertMany(). It is way faster than looping on .create() or providing it with an array.

Usage:

var rawDocuments = [/* ... */];

Book.insertMany(rawDocuments)
    .then(function(mongooseDocuments) {
         /* ... */
    })
    .catch(function(err) {
        /* Error handling */
    });

Or

Book.insertMany(rawDocuments, function (err, mongooseDocuments) { /* Your callback function... */ });

You can track it on:

• https://github.com/Automattic/mongoose/issues/723
• https://github.com/Automattic/mongoose/blob/1887e72694829b62f4e3547283783cebbe66b46b/lib/model.js#L1774

回答3:

Indeed, you can use the "create" method of Mongoose, it can contain an array of documents, see this example:

Candy.create({ candy: 'jelly bean' }, { candy: 'snickers' }, function (err, jellybean, snickers) {
});

The callback function contains the inserted documents. You do not always know how many items has to be inserted (fixed argument length like above) so you can loop through them:

var insertedDocs = [];
for (var i=1; i<arguments.length; ++i) {
    insertedDocs.push(arguments[i]);
}

---

Update: A better solution

A better solution would to use Candy.collection.insert() instead of Candy.create() - used in the example above - because it's faster (create() is calling Model.save() on each item so it's slower).

See the Mongo documentation for more information: http://docs.mongodb.org/manual/reference/method/db.collection.insert/

(thanks to arcseldon for pointing this out)

回答4:

You can perform bulk insert using mongoDB shell using inserting the values in an array.

db.collection.insert([{values},{values},{values},{values}]);

回答5:

You can perform bulk insert using mongoose, as the highest score answer. But the example cannot work, it should be:

/* a humongous amount of potatos */
var potatoBag = [{name:'potato1'}, {name:'potato2'}];

var Potato = mongoose.model('Potato', PotatoSchema);
Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Don't use a schema instance for the bulk insert, you should use a plain map object.

回答6:

It seems that using mongoose there is a limit of more than 1000 documents, when using

Potato.collection.insert(potatoBag, onInsert);

You can use:

var bulk = Model.collection.initializeOrderedBulkOp();

async.each(users, function (user, callback) {
    bulk.insert(hash);
}, function (err) {
    var bulkStart = Date.now();
    bulk.execute(function(err, res){
        if (err) console.log (" gameResult.js > err " , err);
        console.log (" gameResult.js > BULK TIME  " , Date.now() - bulkStart );
        console.log (" gameResult.js > BULK INSERT " , res.nInserted)
      });
});

But this is almost twice as fast when testing with 10000 documents:

function fastInsert(arrOfResults) {
var startTime = Date.now();
    var count = 0;
    var c = Math.round( arrOfResults.length / 990);

    var fakeArr = [];
    fakeArr.length = c;
    var docsSaved = 0

    async.each(fakeArr, function (item, callback) {

            var sliced = arrOfResults.slice(count, count+999);
            sliced.length)
            count = count +999;
            if(sliced.length != 0 ){
                    GameResultModel.collection.insert(sliced, function (err, docs) {
                            docsSaved += docs.ops.length
                            callback();
                    });
            }else {
                    callback()
            }
    }, function (err) {
            console.log (" gameResult.js > BULK INSERT AMOUNT: ", arrOfResults.length, "docsSaved  " , docsSaved, " DIFF TIME:",Date.now() - startTime);
    });
}

回答7:

I used async-forEach(link for async-forEach npm package documentation) to achieve the same.

My code snippet is as below.I am getting the documents in the req.body.

var forEach = require('async-foreach').forEach;    
exports.save_Ctrl = function (req, res) {    
//  var l=req.body;
//  console.log("length:",l.length);

 forEach(req.body, function(item, index, object,err) {

    console.log(req.body[index]);
    var post = new saveObj(req.body[index]);   

        //save model to MongoDB
    post.save(function (err) {
        if (err) {
            console.log('error saving :' + err.message);
            return err;
        }   
        else {
            console.log("Post saved");
        }
    });       

  });    
 }

回答8:

Here are both way of saving data with insertMany and save

1) Mongoose save array of documents with insertMany in bulk

/* write mongoose schema model and export this */
var Potato = mongoose.model('Potato', PotatoSchema);

/* write this api in routes directory  */
router.post('/addDocuments', function (req, res) {
    const data = [/* array of object which data need to save in db */];

    Potato.insertMany(data)  
    .then((result) => {
            console.log("result ", result);
            res.status(200).json({'success': 'new documents added!', 'data': result});
    })
    .catch(err => {
            console.error("error ", err);
            res.status(400).json({err});
    });
})

2) Mongoose save array of documents with .save()

These documents will save parallel.

/* write mongoose schema model and export this */
var Potato = mongoose.model('Potato', PotatoSchema);

/* write this api in routes directory  */
router.post('/addDocuments', function (req, res) {
    const saveData = []
    const data = [/* array of object which data need to save in db */];
    data.map((i) => {
        console.log(i)
        var potato = new Potato(data[i])
        potato.save()
        .then((result) => {
            console.log(result)
            saveData.push(result)
            if (saveData.length === data.length) {
                res.status(200).json({'success': 'new documents added!', 'data': saveData});
            }
        })
        .catch((err) => {
            console.error(err)
            res.status(500).json({err});
        })
    })
})

回答9:

Sharing working and relevant code from our project:

//documentsArray is the list of sampleCollection objects
sampleCollection.insertMany(documentsArray)  
    .then((res) => {
        console.log("insert sampleCollection result ", res);
    })
    .catch(err => {
        console.log("bulk insert sampleCollection error ", err);
    });