WSO2 ESB HTTP connection was forcibly closed by re

Question

We are using WSO2 ESB to clone and broadcast SOAP requests to all our web services, that are being hosted on IIS. Recently we switched to nhttp transport. It was configured as recommended here but each time after the period of inactivity we encount an exception in wso.log

[HTTP Sender I/O dispatcher-4] ERROR ClientHandler I/O error : An existing connection was forcibly closed by the remote host For : 127.0.0.1:4148
java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
...

The length of inactivity period depends on the 'Connection Time-out value' configured in IIS for the specific web service. Seems like WSO2 ESB hasn't closed all opened socket connections and IIS did it instead, when the time-out period has expired.

Is there is a way to avoid this exception? It doesn't seem to cause some performance drawbacks, but looks quite messy in the error logs.

Our configuration, responsible for threads count and timeouts is as follow:

nhttp.properties file:

http.socket.timeout=120000
nhttp_buffer_size=16384
http.tcp.nodelay=1
http.connection.stalecheck=0
http.block_service_list=false

synapse.properties file:

synapse.threads.core = 40
synapse.threads.max = 100
#synapse.threads.keepalive = 5
#synapse.threads.qlen = 10
#synapse.threads.group = synapse-thread-group
#synapse.threads.idprefix = SynapseWorker

synapse.sal.endpoints.sesssion.timeout.default=600000
synapse.global_timeout_interval=120000 

Answer 1

AFAIU, your back-end service (deployed on IIS) has failed to respond within the specified timeout value. This means, the exception is a valid scenario, as the back-end server closes the socket connection.

I think you should check why the back-end service fails to respond without much delay. Do you know the reason for that?

By the way, I guess you are using ESB 4.6.0? I recommend you to use the latest ESB (version 4.8.0) as it has lot of fixes especially for the pass-through transport.

The pass-through transport performs better than the nhttp transport.

Answer 2

By default WSO2 ESB will use 'KEEPALIVE' for connections. This will result in 'An existing connection was forcibly closed by the remote host' error message when the backend (e.g. IIS) does not use 'KEEPALIVE'. Use the following property to disable 'KEEPALIVE':

<property name="NO_KEEPALIVE" value="true" scope="axis2" type="STRING"/>

It may/may not be intuitively obvious, but this "property" entry is inserted into the "in" sequence of the proxy service for the backend service to which you are connecting. It appeared the author was looking for a more global setting in one of the .properties files. There is, but neither for nhttp.properties nor synapse.properties. It would be the setting in [carbon_home]/repository/conf/passthru-http.properties and set the following to disable keepalive: http.connection.disable.keepalive=true