How can I ensure that my HttpClient 4.1 does not l

Question

My server uses data from an internal web service to construct its response, on a per request basis. I'm using Apache HttpClient 4.1 to make the requests. Each initial request will result in about 30 requests to the web service. Of these, 4 - 8 will end up with sockets stuck in CLOSE_WAIT, which never get released. Eventually these stuck sockets exceed my ulimit and my process runs out of file descriptors.

I don't want to just raise my ulimit (1024), because that will just mask the problem.

The reason I've moved to HttpClient is that java.net.HttpUrlConnection was behaving the same way.

I have tried moving to a SingleClientConnManager per request, and calling client.getConnectionManager().shutdown() on it, but sockets still end up stuck.

Should I be trying to solve this so that I end up with 0 open sockets while there are no running requests, or should I be concentrating on request persistence and pooling?

For clarity I'm including some details which may be relevant:

OS: Ubuntu 10.10

JRE: 1.6.0_22

Language: Scala 2.8

Sample code:

val cleaner = Executors.newScheduledThreadPool(1) 
private val client = {
    val ssl_ctx = SSLContext.getInstance("TLS")
    val managers = Array[TrustManager](TrustingTrustManager)
    ssl_ctx.init(null, managers, new java.security.SecureRandom())
    val sslSf = new org.apache.http.conn.ssl.SSLSocketFactory(ssl_ctx, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
    val schemeRegistry = new SchemeRegistry()
    schemeRegistry.register(new Scheme("https", 443, sslSf))
    val connection = new ThreadSafeClientConnManager(schemeRegistry)
    object clean extends Runnable{ 
        override def run = {
            connection.closeExpiredConnections
            connection.closeIdleConnections(30, SECONDS)
        }
    }
    cleaner.scheduleAtFixedRate(clean,10,10,SECONDS)
    val httpClient = new DefaultHttpClient(connection)
    httpClient.getCredentialsProvider().setCredentials(new AuthScope(AuthScope.ANY), new UsernamePasswordCredentials(username,password))
    httpClient
}
val get = new HttpGet(uri)
val entity = client.execute(get).getEntity
val stream = entity.getContent
val justForTheExample = IOUtils.toString(stream)
stream.close()

Test: netstat -a | grep {myInternalWebServiceName} | grep CLOSE_WAIT

(Lists sockets for my process that are in CLOSE_WAIT state)

Post comment discussion:

This code now demonstrates correct usage.

Answer 1

I've marked oleg's answer as correct, as it highlights an important usage point about HttpClient's connection pooling.

To answer my specific original question, though, which was "Should I be trying to solve for 0 unused sockets or trying to maximize pooling?"

Now that the pooling solution is in place and working correctly the application throughput has increased by about 150%. I attribute this to not having to renegotiate SSL and multiple handshakes, instead reusing persistent connections in accordance with HTTP 1.1.

It is definitely worth working to utilize pooling as intended, rather than trying to hack around with calling ThreadSafeClientConnManager.shutdown() after each request etcetera. If, on the other hand, you were calling arbitrary hosts and not reusing routes the way I am you might easily find that it becomes necessary to do that sort of hackery, as the JVM might surprise you with the long life of CLOSE_WAIT designated sockets if you're not garbage collecting very often.

Answer 2

I had the same issue and solved it using the suggesting found here: here. The author touches on some TCP basics:

When a TCP connection is about to close, its finalization is negotiated by both parties. Think of it as breaking a contract in a civilized manner. Both parties sign the paper and it’s all good. In geek talk, this is done via the FIN/ACK messages. Party A sends a FIN message to indicate it wants to close the socket. Party B sends an ACK saying it received the message and is considering the demand. Party B then cleans up and sends a FIN to Party A. Party A responds with the ACK and everyone walks away.

The problem comes in when B doesn’t send its FIN. A is kinda stuck waiting for it. It has initiated its finalization sequence and is waiting for the other party to do the same.

He then mentions RFC 2616, 14.10 to suggest setting up an http header to solve this issue:

postMethod.addHeader("Connection", "close");

Honestly, I don't really know the implications of setting this header. But it did stop CLOSE_WAIT from happening on my unit tests.

Answer 3

One needs to pro-actively evict expired / idle connections from the connection pool, as in the blocking I/O model connections cannot react to I/O events unless they are being read from / written to. For details see

http://hc.apache.org/httpcomponents-client-dev/tutorial/html/connmgmt.html#d4e631