Can I reduce the space of my cache memory?

Question

I am a newbie on GridGain and I have a problem with the heap size on my cache.

I have a CSV file with 2.000.000 ~ 3.000.000 lines, the size of the file is 600~800 megabytes, but when I load this data into my cache on GridGain I need 5~7 gigabytes. Can I reduce the size of the data in the cache? Or do I need 5~7 gigabytes to save a file with 600 megabytes?

Answer 1

Here are the things that can affect memory consumption:

1. Number of backups configured. If you have, for example, 1 backup configured, then you double the amount of memory needed on the cluster.
2. Indexes - if you are creating indexes for your data, indexes do require additional space.
3. If objects are stored in both, serialized and deserialized form, then extra space is required.

I think a good test would be to load the same data into a ConcurrentHashMap and see how much memory would be required for that. I would say, that in a distributed transactional cache like GridGain there is space taken up by versioning, locking, etc.. so the amount of memory required by a distributed cache should be about 2x or 3x of the space required by a simple ConcurrentHashMap.