Has anyone used an object database with a large am

Question

Object databases like MongoDB and db4o are getting lots of publicity lately. Everyone that plays with them seems to love it. I'm guessing that they are dealing with about 640K of data in their sample apps.

Has anyone tried to use an object database with a large amount of data (say, 50GB or more)? Are you able to still execute complex queries against it (like from a search screen)? How does it compare to your usual relational database of choice?

I'm just curious. I want to take the object database plunge, but I need to know if it'll work on something more than a sample app.

Answer 1

Perhaps worth a mention.

The European Space Agency's Planck mission is running on the Versant Object Database. http://sci.esa.int/science-e/www/object/index.cfm?fobjectid=46951

It is a satelite with 74 onboard sensors launched last year which is mapping the infrarred spectrum of the universe and storing the information in a map segment model. It has been getting a ton of hype these days because of it's producing some of the coolest images ever seen of the universe.

Anyway, it has generated 25T of information stored in Versant and replicated across 3 continents. When the mission is complete next year, it will be a total of 50T

Probably also worth noting, object databases tend to be a lot smaller to hold the same information. It is because they are truly normalized, no data duplication for joins, no empty wasted column space and few indexes rather than 100's of them. You can find public information about testing ESA did to consider storage in multi-column relational database format -vs- using a proper object model and storing in the Versant object database. THey found they could save 75% disk space by using Versant.

Here is the implementation: http://www.planck.fr/Piodoc/PIOlib_Overview_V1.0.pdf

Here they talk about 3T -vs- 12T found in the testing http://newscenter.lbl.gov/feature-stories/2008/12/10/cosmic-data/

Also ... there are benchmarks which show Versant orders of magnitude faster on the analysis side of the mission.

CHeers, -Robert

Answer 2

Someone just went into production with a 12 terabytes of data in MongoDB. The largest I knew of before that was 1 TB. Lots of people are keeping really large amounts of data in Mongo.

It's important to remember that Mongo works a lot like a relational database: you need the right indexes to get good performance. You can use explain() on queries and contact the user list for help with this.

Answer 3

Here's some benchmarks on db4o:

http://www.db4o.com/about/productinformation/benchmarks/

I think it ultimately depends on a lot of factors, including the complexity of the data, but db4o seems to certainly hang with the best of them.

Answer 4

You should read the MongoDB use cases. People who are just playing with technology are often just looking at how does this work and are not at the point where they can understand the limitations. For the right sorts of datasets and access patterns 50GB is nothing for MongoDB running on the right hardware.

These non-relational systems look at the trade-offs which RDBMs made, and changed them a bit. Consistency is not as important as other things in some situations so these solutions let you trade that off for something else. The trade-off is still relatively minor ms or maybe secs in some situations.

It is worth reading about the CAP theorem too.

Answer 5

MongoDB powers SourceForge, The New York Times, and several other large databases...

Answer 6

When I started db4o back in 2000 I didn't have huge databases in mind. The key goal was to store any complex object very simply with one line of code and to do that good and fast with low ressource consumption, so it can run embedded and on mobile devices.

Over time we had many users that used db4o for webapps and with quite large amounts of data, going close to todays maximum database file size of 256GB (with a configured block size of 127 bytes). So to answer your question: Yes, db4o will work with 50GB, but you shouldn't plan to use it for terabytes of data (unless you can nicely split your data over multiple db4o databases, the setup costs for a single database are negligible, you can just call #openFile() )

db4o was acquired by Versant in 2008, because it's capabilites (embedded, low ressource-consumption, lightweight) make it a great complimentary product to Versant's high-end object database VOD. VOD scales for huge amounts of data and it does so much better than relational databases. I think it will merely chuckle over 50GB.