Dealing with a large database in Android

Question

I'm currently developing an app that has the potential to create a very large database. I had planned on installing the app on the SD card to allow for some extra room. Recent dealings here have warned me that that might not be a good idea. Are there any steps I can take to mitigate the danger of this course of action? Or are there any better alternatives?

Answer 1

As mentioned above, you can use Berkeley DB (not the Java Edition) on Android. I described the steps of the cross-compiling process here since it requires minor tweaking.

Answer 2

This is a comprehensive post on the subject (I'm not the author).

I think, overall, it needs to be communicated that SQLite is just a SQL mechanism for accessing a file. It appears that the current market limit is 50mb for the entire APK. When installing to internal memory, you require 2x your APK size. Installing to sdcard requires just the stated APK size.

Here is what you will be working against: 1.) Since SQLite is just a abstraction over your file, when you do selects, inserts, updates, etc, you will be incurring sdcard read write costs 2.) I've seen mention of a soft limit of 10000 records based on performance. This article is a bit old, so its likely gotten better.

Other then that, you'll probably have to set up some tests to see what is feasible. Cursory search of google did not show any benchmarks to date.

Answer 3

As pointed out previously, SQLite has the functionality that you're looking for, packaged in a small library. It's designed to replace simple file access with reliable, recoverable, transactional data access via a SQL API. There's a great summary on their main page.

There are literally thousands of projects that are using SQLite. If your data set is going to be very large (more than 100-200MB), then you might want to consider using Berkeley DB as an option. Berkeley DB recently introduced support for a SQL API, which is completely SQLite compatible. In addition to the functionality that's provided by the SQLite SQL parser, query planner and executor, you also get the reliability and scalability that Berkeley DB is well known for. We have several customers who happily started out with SQLite. When they realized that they needed additional concurrency, scalability and reliability that not available in SQLite, they replaced the SQLite library with the BDB library, recompiled their application and had it tested and running on Berkeley within a few days.

I'm one of the Product Managers for Berkeley DB, so I'm a little biased. :-) However, we implemented the BDB SQL API so that we could offer users the best of both worlds: the ubiquity and ease-of-use of SQLite combined with the concurrency, reliability and scalability of Berkeley DB. Especially with larger data sets, Berkeley DB can make all the difference in application performance.