Is there an log option that can let tomcat log the bad query instead just throwing this ?
SEVERE: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
(Tried log level to FULL, but only capture the above)
This is not enough information to further debug
Alternatively, if this can be fixed by allocated more memory by tweaking the following?
-Xms1024M -Xmx4096M -XX:MaxPermSize=256M
Update
-Xms6G -Xmx6G -XX:MaxPermSize=1G -XX:PermSize=512M
(the above seems works better, keep monitoring)
I suspect you might be using sorts on a large index. That's one thing I definitely know can require a large array size with Lucene. Either way, you might want to try using a 64-bit JVM with these options:
-Xmx6G -XX:MaxPermSize=128M -XX:+UseCompressedOops
The last option will reduce 64-bit memory pointers to 32-bit (as long the heap is under 32GB). This typically reduces the memory overhead by about 40%, so it can help stretch your memory significantly.
Update: Most likely you don't need such a large permanent generation size, certainly not 1G. You're probably fine with 128M, and you'll get a specific error if you go over with Java 6. Since you're limited to 8G in your server you might be able to get away with 7G for the heap with a smaller perm gen. Be careful about not going into swap, that can seriously slow things down for Java.
I noticed you didn't mention -XX:+UseCompressedOops in your update. That can make a huge difference if you haven't tried it yet. You might be able to squeeze a little more space out by reducing the size of eden to give the tenured generation more room. Beyond that I think you'll simply need more memory or fewer sort fields.
If you want to find out what causes OutOfMemory, you can add
-XX:+HeapDumpOnOutOfMemoryError
to your java opts.
The next time you get out of memory, you will get a heap dump file that can be analyzed with "jhat" that is located inside jdk/lib. Jhat will show you what objects exist in your heap and how much memory they consume.
You will get this exception because you are trying to create an Array that is larger than the maximum contiguous block of memory in your Java VMs heap.
https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit
What is the solution?
The java.lang.OutOfMemoryError: Requested array size exceeds VM limit can appear as a result of either of the following situations:
Your arrays grow too big and end up having a size between the platform limit and the Integer.MAX_INT
You deliberately try to allocate arrays larger than 2^31-1 elements to experiment with the limits.
In the first case, check your code base to see whether you really need arrays that large. Maybe you could reduce the size of the arrays and be done with it. Or divide the array into smaller bulks and load the data you need to work with in batches fitting into your platform limit.
In the second case – remember that Java arrays are indexed by int. So you cannot go beyond 2^31-1 elements in your arrays when using the standard data structures within the platform. In fact, in this case you are already blocked by the compiler announcing “error: integer number too large” during compilation.
But if you really work with truly large data sets, you need to rethink your options. You can load the data you need to work with in smaller batches and still use standard Java tools, or you might go beyond the standard utilities. One way to achieve this is to look into the sun.misc.Unsafe class. This allows you to allocate memory directly like you would in C.
Upgrade solr to a newer version seems have sort this problem, likely newer version has a better heap memory management.
I use this in catalina.sh
JAVA_OPTS="-Dsolr.solr.home=/etc/tomcat6/solr -Djava.awt.headless=true -server -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+DisableExplicitGC"
I never had mem problems on Tomcat/solr with 30M small documents. I had problems with the solrJ indexing client though. I had to use -Xms8G -Xmx8G for the Java client, and add documents by chunks of 250K documents.
Out of memory! See if there is an array out of bounds, or loop the system resources are swallowed up!
java.lang.OutOfMemoryError: Java heap space
In the JVM, if 98% of the time is available for the GC Heap size and less than 2% of the time to throw this exception information.
JVM heap setting is the java program is running JVM memory space can be used to deploy the settings. JVM at startup automatically set Heap size value, the initial space (ie-Xms) is the physical memory of 1 / 64 , The maximum space (-Xmx) is the physical memory of 1 / 4. JVM can be used to provide the-Xmn-Xms-Xmx and other options can be set.
Requested array size exceeds VM limit: This is because the application of the array size exceeds the size of heap space, such as a 256M of heap space in the array to apply for a 512M
