Give more memory to my jar file

Question

I have a multithreaded crawler. In this program, if I load a lot of seeds, I get an error. I saw the java.lang.OutOfMemoryError and thought maybe the memory is not enough. I tried running the crawler.jar file with these arguments: java -Xms512m -Xmx3G -jar crawler.jar but so far, no luck.

This is the StackTrace of the program:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at com.sleepycat.je.utilint.DaemonThread.runOrPause(DaemonThread.java:99)
        at com.sleepycat.je.dbi.EnvironmentImpl.runOrPauseDaemons(EnvironmentImpl.java:772)
        at com.sleepycat.je.dbi.EnvironmentImpl.envConfigUpdate(EnvironmentImpl.java:717)
        at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:579)
        at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:204)
        at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:230)
        at com.sleepycat.je.Environment.<init>(Environment.java:212)
        at com.sleepycat.je.Environment.<init>(Environment.java:166)
        ...

Is this related to memory as I suspected? Does adding the -Xms512m -Xmx3G work when I'm running the jar file using java -jar?

I ran task manager (I'm running on Windows Server) but after running the app, the memory doesn't go that far higher! Am I wrong?

Answer 1

These kind of errors can often be solved with some simple profiling. The JDK ships with jvisualvm.

E.g. C:\Program Files\Java\jdk1.7.0_11\bin\jvisualvm.exe

You'll be able to see a graph of your heap size and it's also worth checking the PermGen space.

If this dosen't reveal anything, try adding -XX:-HeapDumpOnOutOfMemoryError when you run your jar and then load the heap dump into Eclipse Memory Analyzer.

Answer 2

The -Xms512m -Xmx3G option only effect the heap size of JVM and would not solve your problem.

However, the default thread number limits should be enough in most scenarios. You can increase the limit by tweaking JVM/System options, but no matter how many thread you create, the capacity of your system is bounded to your computer resource. e.g. cpu, memory, network, etc.

I suggest to solve this problem from a different view:

• Try to share the Environment instances among the worker threads.
• Try to control the concurrent level to a rate in which your system is busy and thread number limits is not touched.

Answer 3

You are spawning more threads than Java can take. Try shrinking the stack size with this JVM parameter: -Xss128k

If that isn't it, maybe you are hitting on OS limit. If you are using Linux add a line like this to ~/.profile: ulimit -u 4096

There are many reasons why this might occur depending on your code and the OS you use.