Strategy for debugging surefire “The forked VM ter

Question

I am working on a rather complex java project with many dependencies and many unit tests.

I am using java 1.6.0_65 on mac (mavericks) with maven 3.0.5 with maven-surefire-plugin:2.16 running in several forks. My problem is that running this setup with several forks causes a fork to exit with:

"The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"

running this with only one fork does not produce the problem (and everything passes)

There is some information out there about this problem including this StackOverflow question and this surefire bug (which seems to be solved by now)

I am aware that the "Answer" to this situation is to find what in my code calls System.exit() - Nothing i could find.

Or what causes my JVM to crash - there are no hs_pid crash reports.

My Question is what what kind of strategy can i use to find a locate this cause? To clarify, I am not interested in the answer noted above, but a way to find where it is originating from. (Or even better a different answer altogether to what might be causing this )

---

my Surefire configuration is: (but i did try other combinations)

<parallel>classes</parallel>
<threadCount>1</threadCount>
<forkCount>1C</forkCount>
<reuseForks>false</reuseForks>
<useSystemClassLoader>false</useSystemClassLoader>
<useManifestOnlyJar>true</useManifestOnlyJar>
<useFile>true</useFile>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<runOrder>alphabetical</runOrder>

---

Update #1 adding the relevant out put after running the maven goal with --debug (-X)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.apache.maven.plugin.MojoFailureException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.SurefirePlugin.assertNoException(SurefirePlugin.java:198)
    at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:188)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:852)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:720)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 25 more
Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:316)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:169)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:958)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:822)
    ... 28 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:300)
    ... 31 more
Caused by: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:485)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:352)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$300(ForkStarter.java:85)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:288)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:283)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Answer 1

If anyone is including a custom argLine argument, you must reconsider because it is likely the source of your issues with the memory allocation.

For Example (I used to have):

<argLine>XX:MaxPermSize=4096m ${argLine}</argLine>

Now I use hard specified values:

<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>

For whatever reason, Applications that integrate with Surefire such as Jacoco, dont request enough memory to coexist with the testing that happens at build time.

(more information can also be found at this s.o. question (1))

(1)-maven jacoco: not generating code coverage report

Answer 2

Steps:

(1) Run mvn with the -e and -X options to get more debug information.

(2) Look for "Error" in the output. In my case, when I ran the mvn command, part of the output included:

[ERROR] Command wascmd.exe /X /C "C:\dev\dev-tools\.....

(3) Execute the problematic command directly in the command shell.

In my case, executing

cmd.exe /X /C "C:\dev\dev-tools\....

from the command line resulted in an OutOfMemoryError.

Answer 3

Following Maven Surefire documentation, you can execute the forked VM in debug mode, if it always fails. You can then debug your code until it exit.

Answer 4

FWIW, I have experienced this error when the JVM ran out of memory during the maven build. On linux this was detected by OOM killer, which ended up generating kernel messages like Aug 28 20:53:27 ip-xxx-xxx-xxx-xxx kernel: [248686.775455] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0.

I guess on a mac you'd just want to monitor your memory usage with ActivityMonitor.

Answer 5

What you may want to check is the argline setting for surefire or failsafe in build/pluging/plugin config in your pom. I had something improper in there that caused the forked vm to fail (Ironically, i put maven.failsafe.debug in there to help debug an earlier forking crash).

Answer 6

I just delete all the maven repository and run maven clean install. Then the issue is gone.