I am working on an application which reads in a huge amount of data from a database into a Map<String,Map<String,Map<String,String>>>, processes it, and writes the processed reports to a spreadsheet using an in-house xml writer. The whole run can take about 12 hours.
I'm finding I'm getting
Exception in thread "CursorController-Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
at java.lang.StringBuilder.<init>(StringBuilder.java:68)
When I attempt to write this jumbo file. For this reason I think it would be best to write each Map<String,Map<String,String>> (notice that's a layer deeper)as it's finished processing.
My question is, how can I make sure that the Map<String,Map<String,String>> is not retained in memory after I write it, since the Map>> will still contain it?
Once you're done with the Map<String,Map<String,String>> mapped to by the key "key" you simply do
hugeMap.remove("key");
This will "null" out the entry in the hugeMap and make the Map<String,Map<String,String>> eligible for garbage collection (i.e., never be part of causing a heap space out of memory).
I would choose a different solution for this kind of problem. 12 hours for processing the source data is heavy.
Have you considered any scalable solutions? For e.g. Hadoop?
Use map.remove(key) method on your Map>>. You can call from time to time System.gc(); to force garbage collection.
You can keep the Map> that was written in the outer map if you want to keep your structure. but probably you should clear its contents so that it is empty. also, make sure that when you are processing it and writing it, you don't keep references to its members (mappings) anywhere before clearing the content. please see the following post for picking the approach that suits your needs best Using .clear() or letting the GC take care of it
You can't.
The garbage collector runs whenever it likes and frees whatever it likes.
That said, it is worth trying that after you delete all references to the data you no longer need, call to System.gc().
Anyway, you have written that the out of memory error is while writting the data. Maybe you have a memory leak there.
