How to load 100 million records into MongoDB with

I have a small script that's written in Scala which is intended to load a MongoDB instance up with 100,000,000 sample records. The idea is to get the DB all loaded, and then do some performance testing (and tune/re-load if necessary).

The problem is that the load-time per 100,000 records increases pretty linearly. At the beginning of my load process it took only 4 seconds to load those records. Now, at nearly 6,000,000 records, it's taking between 300 and 400 seconds to load the same amount (100,000)! That's two orders of magnitude slower! Queries are still snappy, but at this rate, I'll never be able to load the amount of data that I'd like.

Will this work faster if I write a file out with all of my records (all 100,000,000!), and then use mongoimport to import the whole thing? Or are my expectations too high and I'm using the DB beyond what it's supposed to handle?

Any thoughts? Thanks!

Here's my script:

import java.util.Date

import com.mongodb.casbah.Imports._
import com.mongodb.casbah.commons.MongoDBObject

object MongoPopulateTest {
  val ONE_HUNDRED_THOUSAND = 100000
  val ONE_MILLION          = ONE_HUNDRED_THOUSAND * 10

  val random     = new scala.util.Random(12345)
  val connection = MongoConnection()
  val db         = connection("mongoVolumeTest")
  val collection = db("testData")

  val INDEX_KEYS = List("A", "G", "E", "F")

  def main(args: Array[String]) {
    populateCoacs(ONE_MILLION * 100)
  }

  def populateCoacs(count: Int) {
    println("Creating indexes: " + INDEX_KEYS.mkString(", "))
    INDEX_KEYS.map(key => collection.ensureIndex(MongoDBObject(key -> 1)))

    println("Adding " + count + " records to DB.")

    val start     = (new Date()).getTime()
    var lastBatch = start

    for(i <- 0 until count) {
      collection.save(makeCoac())
      if(i % 100000 == 0 && i != 0) {
        println(i + ": " + (((new Date()).getTime() - lastBatch) / 1000.0) + " seconds (" +  (new Date()).toString() + ")")
        lastBatch = (new Date()).getTime()
      }
    }

    val elapsedSeconds = ((new Date).getTime() - start) / 1000

    println("Done. " + count + " COAC rows inserted in " + elapsedSeconds + " seconds.")
  }

  def makeCoac(): MongoDBObject = {
    MongoDBObject(
      "A" -> random.nextPrintableChar().toString(),
      "B" -> scala.math.abs(random.nextInt()),
      "C" -> makeRandomPrintableString(50),
      "D" -> (if(random.nextBoolean()) { "Cd" } else { "Cc" }),
      "E" -> makeRandomPrintableString(15),
      "F" -> makeRandomPrintableString(15),
      "G" -> scala.math.abs(random.nextInt()),
      "H" -> random.nextBoolean(),
      "I" -> (if(random.nextBoolean()) { 41 } else { 31 }),
      "J" -> (if(random.nextBoolean()) { "A" } else { "B" }),
      "K" -> random.nextFloat(),
      "L" -> makeRandomPrintableString(15),
      "M" -> makeRandomPrintableString(15),
      "N" -> scala.math.abs(random.nextInt()),
      "O" -> random.nextFloat(),
      "P" -> (if(random.nextBoolean()) { "USD" } else { "GBP" }),
      "Q" -> (if(random.nextBoolean()) { "PROCESSED" } else { "UNPROCESSED" }),
      "R" -> scala.math.abs(random.nextInt())
    )
  }

  def makeRandomPrintableString(length: Int): String = {
    var result = ""
    for(i <- 0 until length) {
      result += random.nextPrintableChar().toString()
    }
    result
  }
}

Here's the output from my script:

Creating indexes: A, G, E, F
Adding 100000000 records to DB.
100000: 4.456 seconds (Thu Jul 21 15:18:57 EDT 2011)
200000: 4.155 seconds (Thu Jul 21 15:19:01 EDT 2011)
300000: 4.284 seconds (Thu Jul 21 15:19:05 EDT 2011)
400000: 4.32 seconds (Thu Jul 21 15:19:10 EDT 2011)
500000: 4.597 seconds (Thu Jul 21 15:19:14 EDT 2011)
600000: 4.412 seconds (Thu Jul 21 15:19:19 EDT 2011)
700000: 4.435 seconds (Thu Jul 21 15:19:23 EDT 2011)
800000: 5.919 seconds (Thu Jul 21 15:19:29 EDT 2011)
900000: 4.517 seconds (Thu Jul 21 15:19:33 EDT 2011)
1000000: 4.483 seconds (Thu Jul 21 15:19:38 EDT 2011)
1100000: 4.78 seconds (Thu Jul 21 15:19:43 EDT 2011)
1200000: 9.643 seconds (Thu Jul 21 15:19:52 EDT 2011)
1300000: 25.479 seconds (Thu Jul 21 15:20:18 EDT 2011)
1400000: 30.028 seconds (Thu Jul 21 15:20:48 EDT 2011)
1500000: 24.531 seconds (Thu Jul 21 15:21:12 EDT 2011)
1600000: 18.562 seconds (Thu Jul 21 15:21:31 EDT 2011)
1700000: 28.48 seconds (Thu Jul 21 15:21:59 EDT 2011)
1800000: 29.127 seconds (Thu Jul 21 15:22:29 EDT 2011)
1900000: 25.814 seconds (Thu Jul 21 15:22:54 EDT 2011)
2000000: 16.658 seconds (Thu Jul 21 15:23:11 EDT 2011)
2100000: 24.564 seconds (Thu Jul 21 15:23:36 EDT 2011)
2200000: 32.542 seconds (Thu Jul 21 15:24:08 EDT 2011)
2300000: 30.378 seconds (Thu Jul 21 15:24:39 EDT 2011)
2400000: 21.188 seconds (Thu Jul 21 15:25:00 EDT 2011)
2500000: 23.923 seconds (Thu Jul 21 15:25:24 EDT 2011)
2600000: 46.077 seconds (Thu Jul 21 15:26:10 EDT 2011)
2700000: 104.434 seconds (Thu Jul 21 15:27:54 EDT 2011)
2800000: 23.344 seconds (Thu Jul 21 15:28:17 EDT 2011)
2900000: 17.206 seconds (Thu Jul 21 15:28:35 EDT 2011)
3000000: 19.15 seconds (Thu Jul 21 15:28:54 EDT 2011)
3100000: 14.488 seconds (Thu Jul 21 15:29:08 EDT 2011)
3200000: 20.916 seconds (Thu Jul 21 15:29:29 EDT 2011)
3300000: 69.93 seconds (Thu Jul 21 15:30:39 EDT 2011)
3400000: 81.178 seconds (Thu Jul 21 15:32:00 EDT 2011)
3500000: 93.058 seconds (Thu Jul 21 15:33:33 EDT 2011)
3600000: 168.613 seconds (Thu Jul 21 15:36:22 EDT 2011)
3700000: 189.917 seconds (Thu Jul 21 15:39:32 EDT 2011)
3800000: 200.971 seconds (Thu Jul 21 15:42:53 EDT 2011)
3900000: 207.728 seconds (Thu Jul 21 15:46:21 EDT 2011)
4000000: 213.778 seconds (Thu Jul 21 15:49:54 EDT 2011)
4100000: 219.32 seconds (Thu Jul 21 15:53:34 EDT 2011)
4200000: 241.545 seconds (Thu Jul 21 15:57:35 EDT 2011)
4300000: 193.555 seconds (Thu Jul 21 16:00:49 EDT 2011)
4400000: 190.949 seconds (Thu Jul 21 16:04:00 EDT 2011)
4500000: 184.433 seconds (Thu Jul 21 16:07:04 EDT 2011)
4600000: 231.709 seconds (Thu Jul 21 16:10:56 EDT 2011)
4700000: 243.0 seconds (Thu Jul 21 16:14:59 EDT 2011)
4800000: 310.156 seconds (Thu Jul 21 16:20:09 EDT 2011)
4900000: 318.421 seconds (Thu Jul 21 16:25:28 EDT 2011)
5000000: 378.112 seconds (Thu Jul 21 16:31:46 EDT 2011)
5100000: 265.648 seconds (Thu Jul 21 16:36:11 EDT 2011)
5200000: 295.086 seconds (Thu Jul 21 16:41:06 EDT 2011)
5300000: 297.678 seconds (Thu Jul 21 16:46:04 EDT 2011)
5400000: 329.256 seconds (Thu Jul 21 16:51:33 EDT 2011)
5500000: 336.571 seconds (Thu Jul 21 16:57:10 EDT 2011)
5600000: 398.64 seconds (Thu Jul 21 17:03:49 EDT 2011)
5700000: 351.158 seconds (Thu Jul 21 17:09:40 EDT 2011)
5800000: 410.561 seconds (Thu Jul 21 17:16:30 EDT 2011)
5900000: 689.369 seconds (Thu Jul 21 17:28:00 EDT 2011)

回答1:

Some tips :

Do not index your collection before inserting, as inserts modify the index which is an overhead. Insert everything, then create index .
instead of "save" , use mongoDB "batchinsert" which can insert many records in 1 operation. So have around 5000 documents inserted per batch. You will see remarkable performance gain .

see the method#2 of insert here, it takes array of documents to insert instead of single document. Also see the discussion in this thread

And if you want to benchmark more -
This is just a guess, try using a capped collection of a predefined large size to store all your data. Capped collection without index has very good insertion performance.

回答2:

I've had the same thing. As far as I can tell, it comes down to the randomness of the index values. Whenever a new document is inserted, it obviously also needs to update all the underlying indexes. Because you're inserting random, as opposed to sequential, values into these indexes, you're constantly accessing the entire index to find where to place the new value.

This is all fine to begin with when all the indexes are sitting happily in memory, but as soon as they grow too large you need to start hitting the disk to do index inserts, then the disk starts thrashing and write performance dies.

As you're loading the data, try comparing db.collection.totalIndexSize() with the available memory, and you'll probably see this happen.

Your best bet is to create the indexes after you've loaded the data. However, this still doesn't solve the problem when it's the required _id index that contains a random value (GUID, hash, etc.), then your best approach might be to think about sharding or getting more RAM.

回答3:

What I did in my project was adding up a bit of multithreading (the project is in C# but I hope the code is self-explanatory). After playing with the necessary number of threads it turned out that setting the number of threads to the number of cores leads to a slightly better performance(10-20%) but I suppose this boost is hardware specific. Here is the code:

    public virtual void SaveBatch(IEnumerable<object> entities)
    {
        if (entities == null)
            throw new ArgumentNullException("entities");

        _repository.SaveBatch(entities);
    }


    public void ParallelSaveBatch(IEnumerable<IEnumerable<object>> batchPortions)
    {
        if (batchPortions == null)
            throw new ArgumentNullException("batchPortions");
        var po = new ParallelOptions
                 {
                     MaxDegreeOfParallelism = Environment.ProcessorCount
                 };
        Parallel.ForEach(batchPortions, po, SaveBatch);
    }

回答4:

Another alternative is to try TokuMX. They use Fractal Indexes which means that it does not slow down over time as the database gets bigger.

TokuMX is going to be included as a custom storage driver in an upcoming version of MongoDB.

The current version of MongoDB runs under Linux. I was up and running on Windows quite quickly using Vagrant.