WAITING threads issue in wso2esb

Question

Hi am working on wso2esb and using Active MQ for message queues.

After around 3 weeks of usage ESB server was hanging and with the help of JMX monitoring of ESB i found that they are a huge number of java threads are in WAITING state.

 [EsbMonitoring] ***************** java  Threads Attributes *********************
 [EsbMonitoring] ThreadCount              :8873
 [EsbMonitoring] DaemonThreadCount        :104
 [EsbMonitoring] PeakThreadCount          :8992
 [EsbMonitoring] TotalStartedThreadCount  :16086123

Initially when we start the ESB server they are around 560 threads

 [EsbMonitoring] ***************** java  Threads Attributes *********************
 [EsbMonitoring] ThreadCount              :592
 [EsbMonitoring] DaemonThreadCount        :78
 [EsbMonitoring] PeakThreadCount          :592
 [EsbMonitoring] TotalStartedThreadCount  :1510

I took thread dump with jstack of ESB server with

"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60595" prio=10 tid=0x00007fdb1c890800 nid=0x259e waiting on condition [0x00007fda951cd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000070bdf6c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60589" prio=10 tid=0x00007fdb1c8b7800 nid=0x259a waiting on condition [0x00007fda950cc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000070be68c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60590" prio=10 tid=0x00007fdb1c8ae800 nid=0x2597 waiting on condition [0x00007fda946c2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000070be22c88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

All this connections are closed in the active MQ side but ESB is still holding these threads and troubling ESB and making consumption high and after reaching curtain point it hangs . I need to restart the ESB server to solve this issue.

Firstly why are these connection are WAITING state forever and why ESB is not removing these kind of threads , Is there any way i can kill or destroy WAITING threads with out restarting ESB.

Any help would be appreciated.

Thank You..!

Answer 1

Another few questions

• What version of WSO server ?
• What is jvm version ?

Anyway, i assume you have pretty high load. I don't think i can reproduce it easy in short time. I had something similar with postgresql jdbc driver, connections were getting stuck after some time, and all connection in pool are got blocked. It was fixed just on jdbc driver update.

According http://mvnrepository.com/artifact/org.apache.activemq/activemq-broker version 5.8.0 quite old. It was released in 2013. There are some bug reports in activemq jira about connection leak. Few thing you can try.

Get latest version of library from jboss repository. Link you will find on same page as shown on picture underneath

Just be accurate with library dependencies. Update dependencies as well. List of jar files is underneath (It is from official documentation https://docs.wso2.com/display/ESB500/Configure+with+ActiveMQ)

• activemq-broker-5.8.0.jar
• activemq-client-5.8.0.jar
• activemq-kahadb-store-5.8.0.jar
• geronimo-jms_1.1_spec-1.1.1.jar
• geronimo-j2ee-management_1.1_spec-1.0.1.jar
• geronimo-jta_1.0.1B_spec-1.0.1.jar
• hawtbuf-1.9.jar
• slf4j-api-1.6.6.jar
• activeio-core-3.1.4.jar (available in /lib/optional folder)

Another way just update to next release of activemq broker. ActiveMq mature project, so it less possible that in every new release it introduces massive amount of new features and break compatibility with previous version. If release notes doesn't state anything extra ordinary, try it, test it in test environment and make decision.