I have a service that is expected to execute requests at ~5 or more requests/min. This service depends on Apache AsyncHttpClient. After every few minutes, the clients hits some condition which causes java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED. All requests to the client start failing with same exception message. After service is restarted, this cycle repeats.
It is really hard to debug this problem as the request execution failure surprisingly does not cause a callback to the failed() method of the AsyncResponse.
From what I could gather, there has been a fix HTTPCORE-370 in HttpCore NIO which solved a similar problem in 4.3.2. I am using the following version -
commons-httpclient-3.1.jar
httpasyncclient-4.1.1.jar
httpcore-4.4.4.jar
httpcore-nio-4.4.4.jar
Yet seeing this problem.
I've been dealing with this same exception in my application, and I finally found a helpful suggestion from this post - http://httpcomponents.10934.n7.nabble.com/I-O-reactor-status-STOPPED-td29059.html
You can use #getAuditLog() method of the I/O reactor to find out exactly
what exception caused it to terminate.
If you keep a reference to your ConnectionManager's IOReactor, you can call this method to get insight into the actual problem:
http://hc.apache.org/httpcomponents-core-4.4.x/httpcore-nio/apidocs/org/apache/http/impl/nio/reactor/AbstractMultiworkerIOReactor.html#getAuditLog()
Turns out I was doing something incredibly stupid in my own code. But I couldn't figure it out until I read the audit log.
If you see OutOfMemoryError before this, try this
-XX:MaxDirectMemorySize=512M
See https://issues.apache.org/jira/browse/HTTPASYNC-104
In my case, using Elasticsearch high level client, this exception is due to esclient.indexAsync(indexRequest,RequestOptions.DEFAULT,null)
I fixed it by add an action listeners in all async requests like this
esclient.indexAsync(indexRequest,RequestOptions.DEFAULT,
new ActionListener<IndexResponse>() {
@Override
public void onResponse(IndexResponse response) {
}
@Override
public void onFailure(Exception e) {
});
