How to add more than 50,000,000 records in arrayli

Question

Add into array list more than 50,000,000 records from data base. I am adding 30,000,000 records it showing out of heap memory error .

Answer 1

Use chunking, as it will increase performance and solve your heap memory error. I mean use 100 arralist of 50000 size, in total it will be 5000000.

Answer 2

You shouldn't store so much data an ArrayList. For such reasons where databases invented.

If you want to get a specific data you should execute a query and then store the result as an object, and put these objects into an a List if you need them any more.

If you need to Aggregate Data you could also make a Wrapper objects for tha data.

Answer 3

Increasing your heap size is one option, as Ron points out. Normally when there are a lot of records to be processed, the best practice is to determine a way to process them sequentially so that only one record plus whatever summary data is needed has to be in memory at the same time. In addition, if I need to find the max of a column in a table, or an average of it, I can use a query, etc.

Answer 4

Another consideration is to look at the 50000000 records and see if they can be logically grouped into smaller groups. Then, process each group by itself so you don't run out of memory. Example: All the names in a telephone book can be grouped by the first letter of last name. Therefore all people who's last name begins with 'A' are grouped together and processed first, moving on to last names beginning with 'B'. I think this will a better approach than arbitrarily processing a group of 10,000 records at a time.

If you do decide to put the data in a database, you should carefully design the database schema and not just create a single database table to hold all 50,000,000 records.

Answer 5

Have a look here: Increase heap size in Java

You can set a flag on the JVM to increase the heap size. Use whatever you want but be sure you have the RAM obviously.

From the command line:

java -Xmx8g myprogram
java -Xmx16g myprogram

Or if you know how much space your program needs, set the initial heap size:

java -Xms4g myprogram

Use these cautiously. As others have commented, this is more likely a problem with your approach than the default limit of the Java heap.