JMS connections exhausted using WebSphere MQ

Question

I have configured CachingConnectionFactory that wraps a MQTopicConnectionFactory and MQQueueConnectionFactory with cache size set to 10 each.

These are than used in several jms:outbound-channel-adapter or jms:message-driven-channel-adapter as part of various spring integration workflows that I have in my application.

It is noticed that once in a while the connection count on MQ channel reaches maximum allowed (about 1000) when the process stops functioning. This is a serious problem for a production application.

Bringing the application down does not reduce the connection count so looks like orphaned connections on MQ side? I am not sure if I am missing anything in my spring jms / SI configuration that can resolve this issue, any help would be highly appreciated.

Also I would like to log connection open and close from application but don't see a way to do that.

<bean id="mqQcf" class="com.ibm.mq.jms.MQQueueConnectionFactory">
//all that it needs host/port/ queue manager /channel
</bean>
 
 <bean id="qcf" class="org.springframework.jms.connection.CachingConnectionFactory">
			<property name="targetConnectionFactory" ref=" mqQcf "/>
			<property name="sessionCacheSize" value="10"/>			
</bean>


<bean id="mqTcf" class="com.ibm.mq.jms.MQTopicConnectionFactory">
//all that it needs host/port/ queue manager /channel
</bean>

<bean id="tcf" class="org.springframework.jms.connection.CachingConnectionFactory">
			<property name="targetConnectionFactory" ref=" mqTcf "/>
			<property name="sessionCacheSize" value="10"/>			
</bean>

//Qcf and tcf are than used in spring integration configuration as required

Thanks

Answer 1

I was debugging a similar issue in my application about the number of open output counts in MQ when there is only one Connection is opened by the connection factory.

The number output counts in MQ explorer is the number of connection handles created by the IBM MQ classes. Per IBM documentation, A Session object encapsulates an IBM MQ connection handle, which therefore defines the transnational scope of the session.

Since the session cache size was 10 in my application, there were 10 IBM MQ connections handles created (one for each session) stayed open for days and the handle state was inactive.

More info can be found in,

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q031960_.htm

As Gary Russell mentioned, Spring doesn't provide way to configure the time outs for these idle connections. IBM has inbuilt properties in MQConnectionFactory which can be configured to setup the reconnect timeouts.

More infor can be found in,

https://www.ibm.com/developerworks/community/blogs/messaging/entry/simplify_your_wmq_jms_client_with_automatic_client_reconnection19?lang=en

The reconnect on exception is true by default for CCF. So care should be taken if IBM throws an exception after time out interval. I am not sure if there is a max number of times reconnect will try before throwing an exception in CCF.

Answer 2

You really need to show your configuration but the Spring CachingConnectionFactory only creates a single connection that's shared for all sessions. Turning on INFO logging for the CCF category emits this log when a new connection is created...

if (logger.isInfoEnabled()) {
    logger.info("Established shared JMS Connection: " + this.target);
}

EDIT:

There's nothing in your config that stands out. As I said, each CCF will have at most 1 connection open at a time.

One possibility, if you have idle times, is that the network (a switch or firewall) might be silently dropping connections without telling the client or server. The next time the client tries to use its connection it will fail and create a new one but the server may never find out that the old one is dead.

Typically, for such situations, enabling heartbeats or keepalives would keep the connection active (or at least allow the server to know it's dead).