Best way to save/load large amout of data in a .Ne

Question

What is the best way to save large amount of data for a .Net 4.0 application?

Right now I am using Lists and serializing to a file in "User Data" folder, and its working ok, but I want to know if there is a better/faster way of saving/loading large amount of data.

The data that I am saving contains only lots of words, like documents.

The size of the data is almost 1 mb.

Answer 1

That really depends on the type of your application. I wouldn't use SQL database of any sort for to just load and save operation of data that I do not need to query or transform. The time it will take to map your object graph to a relational model just not worth it. Also I don't believe it will ever be faster than simple serialization due to the overhead associated with databases (connection management and mapping)

My recent experience was with BinnaryFormatter which had excellent results (files ~ 15mb). Worse come to worse you can always write your own formatter.

Answer 2

If you want fast access, one approach would be to serialize to a hashtable, and cache it. In between reads and writes...

Problem here is ofcourse, versioning, changing of namespaces(then you wont be able to deserialize....easyly), deadlocks, concurrency etc....

Better if you save the file as a XML/JSON, and when you do read it in to memory save it into a hashtable...for fast access...

Answer 3

When you say "large amout [sic] of data", what exactly do you mean by that? A megabyte? a terabyte?

And what exactly is the data?

If it's a set of account records, it might well belong in a database of some sort; if it's a set of images or word processing documents, perhaps not.

Answer 4

Kinda depends on your data and how you have it stored in your app.

But all these NoSQL storage systems are a possibility or just plain binary data into a file.