MongoDB using much memory

Question

Im tryin to migrate a mysql table to mongodb. My table has 6 million entries. Im using java with morphia. When i save about 1,2 million my memory is almost all consumed.

I've read that mongo store the data in memory and after save in disk. Is it possible to send something like a commit to free some amount of memory?

Answer 1

1) In terms of durability, you can tell the MongoDB java driver (which Morphia is using), which strategy to use, see https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/WriteConcern.java#L53. It's simply a trade-off between speed: NONE (not even connectivity issues will cause an error) up to FSYNC_SAFE (the data is definitely written to disk). For the internal details check out http://www.kchodorow.com/blog/2012/10/04/how-mongodbs-journaling-works/

2) Your whole data is mapped to memory (that's why the 32bit edition has a size limit of 2GB), however it is only actually loaded, when required. MongoDB leaves that to the operating system by using mmap. So as long as there is more RAM available, MongoDB will happily load all the data it needs into RAM to make queries very quick. If there is no more memory available, it's up to the operating system to swap out old stuff. This has the nice effect that your data will be kept in memory even if you restart the MongoDB process; only if you restart the server itself the data must be fetched from disk again. I think the downside is that the database process might have a slightly better understanding of what should be swapped out first in comparison to the operating system. I'm not using MongoDB on Windows and haven't seen that message on Mac or Linux (yet), but the operating system should handle that for you (and automatically swap out pieces of information as required). Have you tried setting the driver to JOURNAL_SAFE (should be a good compromise between data security and speed)? In that setting, no data should be lost, even if the MongoDB process dies.

3) In general MongoDB is built to use as much available memory as possible, but you might be able to restrict it with http://captaincodeman.com/2011/02/27/limit-mongodb-memory-use-windows/ - which I haven't tested, as we are using (virtual) Linux servers.

Answer 2

if you just want to release some memory mongodb uses, after your data is processed and mongod is idle, you can run this command

use admin
db.runCommand({closeAllDatabases: 1})

then , you will see the mapped,vsize, res that outputed by mongostat go down a lot.

I have try, and it works. Hope to helps , ^_^