Improving JBoss 5 slow startup

Question

We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.

18:53:35,444 INFO  [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms

It seems like the component that is taking JBoss the longest time to initialize is the JMX

18:50:41,926 INFO  [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl@1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO  [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector

From the DEBUG server log, I get these lines at the problematic time:

2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData@1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false

There is no EJB in the project.

The memory settings are:

 -Xms128m -Xmx512m -XX:MaxPermSize=256m

Do you have any idea how JBoss start time can be improved?

Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant

Answer 1

Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.

While I'm no expert, I would suggest the following:

1. Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
   
2. Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.

Answer 2

See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.

Answer 3

A shot into the blue sky without more information

• Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.

• JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname

• System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).

Answer 4

1. Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming

2. When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.

I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.