Disk backed dictionary/cache for c#

Question

I'm looking for a drop in solution for caching large-ish amounts of data.

related questions but for different languages:

• Python Disk-Based Dictionary
• Disk-backed STL container classes?

Close question in different terms:

• Looking for a simple standalone persistant dictionary implementation in C#

I don't need (or want to pay anything for) persistence, transactions, thread safety or the like and want something that is not much more complex to use than a List<> or Dictionary<>.

If I have to write code, I'll just save everything off as files in the temp directory:

string Get(int i)
{
   File.ReadAllText(Path.Combine(root,i.ToString());
}

In my cases in index will be an int (and they should be consecutive or close enough) and the data will be a string so I can get away with treating both a POD and would rather go ultra-light and do exactly that.

The usage is that I have a sequence of 3k files (as in file #1 to #3000) totaling 650MB and need to do a diff for each step in the sequence. I expect that to total about the same or a little more and I don't want to keep all that in memory (larger cases may come along where I just can't).

---

A number of people have suggested different solutions for my problem. However none seem to be targeted at my little niche. The reasons that I'm looking at disk backed caching is because I'm expecting that my current use will use up 1/3 to 1/2 of my available address space. I'm worried that larger cases will just flat run out of space. I'm not worried about treading, persistence or replication. What I'm looking for is a minimal solution using a minimum of code, a minimal usage foot print, minimal in memory overhead and minimum complexity.

I'm starting to think I'm being overly optimistic.

Answer 1

I've partially poprted EhCache Java application to .NET The distributed caching is not yet implemented, but on a single node, all original UnitTests pass. Full OpenSource:

http://sourceforge.net/projects/thecache/

I can create a binary drop if you need it (only sourcecode is availble now)

Answer 2

I recommend the Caching Application block in the Enterprise Library from MS. That was recommended as well, but the link points to an article on the Data Access portion of the Enterprise Library.

Here is the link to the Caching Application Block:

http://msdn.microsoft.com/en-us/library/cc309502.aspx

And specifically, you will want to create a new backing store (if one that persists to disk is not there):

http://msdn.microsoft.com/en-us/library/cc309121.aspx

Answer 3

This is very similar to my question

Looking for a simple standalone persistant dictionary implementation in C#

I don't think a library that exactly fits what you want exists, maybe its time for a new project on github.

Answer 4

I'd take the embedded DB route (SQLite, Firebird), but here are some other options:

• Berkeley DB: not your standard SQL embedded DB, but seems to be the right choice for this kind of task, although not easily usable from .net
• db4o: an OODB, very simple interface