AWS SWF Decider 'java.net.SocketTimeoutExcepti

Question

I'm encountering this error whenever my SWF Worker application polls. If there's an entry in the SWF queue, it gets processed properly. But if it's empty, this timeout exception is returned.

com.amazonaws.http.AmazonHttpClient      : Unable to execute HTTP request: Read timed out

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_31]
        at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.8.0_31]
        at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.8.0_31]
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_31]
        at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_31]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961) ~[na:1.8.0_31]
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:918) ~[na:1.8.0_31]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_31]
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[httpcore-4.4.4.jar!/:4.4.4]
        at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82) ~[aws-java-sdk-core-1.11.22.jar!/:na]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[httpcore-4.4.4.jar!/:4.4.4]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar!/:4.5.1]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.1.jar!/:4.5.1]
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880) [aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723) [aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475) [aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437) [aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386) [aws-java-sdk-core-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.AmazonSimpleWorkflowClient.doInvoke(AmazonSimpleWorkflowClient.java:3133) [aws-java-sdk-simpleworkflow-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.AmazonSimpleWorkflowClient.invoke(AmazonSimpleWorkflowClient.java:3103) [aws-java-sdk-simpleworkflow-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.AmazonSimpleWorkflowClient.pollForDecisionTask(AmazonSimpleWorkflowClient.java:1967) [aws-java-sdk-simpleworkflow-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.flow.worker.DecisionTaskPoller.poll(DecisionTaskPoller.java:171) [aws-java-sdk-swf-libraries-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.flow.worker.DecisionTaskPoller.access$0(DecisionTaskPoller.java:158) [aws-java-sdk-swf-libraries-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.flow.worker.DecisionTaskPoller$DecisionTaskIterator.<init>(DecisionTaskPoller.java:44) [aws-java-sdk-swf-libraries-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.flow.worker.DecisionTaskPoller.pollAndProcessSingleTask(DecisionTaskPoller.java:197) [aws-java-sdk-swf-libraries-1.11.22.jar!/:na]
        at com.amazonaws.services.simpleworkflow.flow.worker.GenericWorker$PollServiceTask.run(GenericWorker.java:94) [aws-java-sdk-swf-libraries-1.11.22.jar!/:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]

What's causing this?

I'm using this setup:

@Workflow
@WorkflowRegistrationOptions(
        defaultExecutionStartToCloseTimeoutSeconds = 259200, //3 days
        defaultTaskStartToCloseTimeoutSeconds = 180
)

Answer 1

It happens when HTTP client socket timeout is less than 61 seconds. Long poll doesn't return up to 60 seconds and java.net.SocketTimeoutException: Read timed out is thrown.

Use ClientConfiguration.setSocketTimeout to change the timeout. This structure is passed to AmazonSimpleWorkflowClient constructor. Looking at AmazonSimpleWorkflowClient implementation it should default to 90 seconds if not set. So make sure that you are using the latest client and the timeout is set correctly.