NetConnection.Call.Failed happening sporadically i

Question

I have a very large problem. I've written a large app using Flex3/Tomcat/BlazeDS/Spring that has worked very well while developing locally, fine when I deployed to a common dev environment, but then fails very often when deployed to our test environment.

The failures seem to happen most when a remoting request takes a good bit of time (more than 20 seconds). On my dev server, the errors happen, but only when the request takes a very long time (more than 45 seconds). However, the error also occurs seemingly instantly at times (send a request and it fails immediately). Most of the errors also have HTTP Status: 502 on them (Bad Gateway), but some just say HTTP Failed.

I haven't done anything special with BlazeDS other than put the war file in the webapps directory. The application is accessed over https. Remoting calls refer to the server by "destination" only. The tomcat servers on our DEV and TEST instances have identical java arguments (same Xms and Xmx, etc). Below are my relevant files:

services-config.xml

<?xml version="1.0" encoding="UTF-8"?>
<services-config>
    <services>
        <default-channels>
           <channel ref="my-secure-amf"/>
        </default-channels>
    </services>
    <channels>
        <channel-definition id="my-secure-amf" class="mx.messaging.channels.SecureAMFChannel">
            <endpoint url="https://{server.name}:{server.port}/{context.root}/messagebroker/amfsecure" class="flex.messaging.endpoints.SecureAMFEndpoint"/>
            <properties>
                <add-no-cache-headers>false</add-no-cache-headers>
            </properties>
        </channel-definition>
    </channels>
<services-config>

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">

    ...

    <servlet>
        <servlet-name>spring-flex</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>/WEB-INF/flexContext.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>spring-flex</servlet-name>
        <url-pattern>/messagebroker/*</url-pattern>
    </servlet-mapping>
</web-app>

flexContext.xml

<!--
    I have component-scan tags here to scan my class files. The Controller I'm using for flex has an annotation on it to define it as a remote destination. Here it is:
    @Controller
    @RemotingDestination(value = "dest", channels="my-secure-amf")
    public class FlexController {
-->

<bean id="flexExceptionTranslator" class="edu.liberty.zconduct.web.FlexExceptionTranslator" />

<flex:message-broker>
    <flex:exception-translator ref="flexExceptionTranslator"/>
    <!-- <flex:secured /> Had this previously, but it wasn't working then, either -->
</flex:message-broker>

I'm near the end of my rope. The error is an app-breaker. Please, please help me if you can.

EDIT I switched over to http and still get the error. Tomcat is now saying:

WARNING: Error sending end packet

java.net.SocketException: Broken pipe

I look in my apache log for jk_mod and see some debug information that looks like the request I sent in, and then the following:

[debug] ajp_send_request::jk_ajp_common.c (1592): Resent the request body (432)

[debug] jk_shutdown_socket::jk_connect.c (681): About to shutdown socket 68

[debug] jk_shutdown_socket::jk_connect.c (732): Shutdown socket 68 and read 0 lingering bytes

[info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (myTestServer) can't receive the response header message from tomcat, network problems or tomcat (xx.xx.xx.xx:xxxx) is down (errno=11)

[error] ajp_get_reply::jk_ajp_common.c (1962): (myTestServer) Tomcat is down or refused connection. No response has been sent to the client (yet)

[info] ajp_service::jk_ajp_common.c (2447): (myTestServer) sending request to tomcat failed (recoverable), (attempt=2)

[error] ajp_service::jk_ajp_common.c (2466): (myTestServer) connecting to tomcat failed.

[debug] ajp_reset_endpoint::jk_ajp_common.c (743): (myTestServer) resetting endpoint with sd = 4294967295 (socket shutdown)

[debug] ajp_done::jk_ajp_common.c (2905): recycling connection pool slot=0 for worker myTestServer

[info] jk_handler::mod_jk.c (2615): Service error=0 for worker=myTestServer

Answer 1

Cornel Creanga helped me arrive at this solution. I switched my app to http and started receiving messages in the catalina.out file. I don't know why this affected it, but it did. The messages all said that there was a broken pipe in the socket. Further investigation showed that the jk_mod was involved.

Through much searching, I discovered that our SysAdmins had set the socket timeout on the server to 10 seconds on our TEST servers, but 5 minutes on our DEV servers, so the massive searches being done were causing it to time out once it got to TEST.

The SysAdmins adjusted the configuration on TEST so that another worker (with a 5 minute timeout) handles the calls for this context path.