Currently I am debugging a seemingly random error with our applications. I will attempt to try to provide as much detail as possible. Basically after a while, one of the application pools for an application on our server stops. I have to start it, or sometimes, restart Coldfusion or worse the machine to get things running again.
The Server logs indicate the pool was stopped because of the Rapid Fail Succession thing in IIS. (The App serves X number of errors in a short amount of time so IIS stops the application)
Going through the ColdFusion Error logs I get this one right around the time of the IIS error where it stops my Pool.
SEVERE: Error in getRealPathFromConn java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:124) at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:391)... AND SO ON
If you really need the full trace I can provide it, but it is long. I am not sure what the getRealPathFromConn is about. I've tried Googling, but haven't come up with a conclusion yet. In the traces they do lead to different files in various applications, but the lines in question seem to have nothing in common with one another.
Any thoughts on what might be the cause or where I should look next would help a ton.
EDIT: I forgot to say, I "believe" this started Monday. The only real thing that changed prior to Monday was the following.
We have an app structure:
root/app1/application.cfc
root/app2/application.cfc
root/app3/application.cfc
root/app3/website1
root/app3/website2
root/app3/website3
I changed the structure so two of the websites have their own application.cfc because we are slowly rolling out a change in how the websites work. I am testing this on two of them for now. Since I made that change, the apppool that keeps crashing the the app pool for app1. Not any of the websites, but I feel like it could be related.
UPDATE Updated Coldfusion to most recent version. (10,0,7,283649) Systems ran for about a week and this morning crashed.
Coldfusion-error.log at that time:
Jan 25, 2013 8:09:11 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\ColdFusion10\\cfusion\lib;C:\\ColdFusion10\\cfusion\jintegra\bin;C:\\ColdFusion10\\cfus ion\jintegra\bin\international;C:\\ColdFusion10\\cfusion\lib\oosdk\classes\win
Jan 25, 2013 8:09:12 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services
Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 20.4-b02
Jan 25, 2013 8:09:29 AM org.apache.catalina.core.ApplicationContext log
INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646
Jan 25, 2013 8:09:35 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2013 8:09:35 AM com.adobe.coldfusion.launcher.Launcher run
INFO: Server startup in 25362 ms
Jan 25, 2013 8:10:04 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer
SEVERE: Error in getRealPathFromConn
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:319)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:445)
at org.apache.coyote.ajp.AjpProcessor.readString(AjpProcessor.java:425)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:407)
at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1154)
at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)
at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)
at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)
at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)
at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:335)
at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:179)
at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)
at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)
at coldfusion.runtime.TemplateProxyFactory.getFullName(TemplateProxyFactory.java:1092)
at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:202)
at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159)
at coldfusion.runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120)
at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:138)
at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:84)
at coldfusion.runtime.AppEventInvoker.<init>(AppEventInvoker.java:64)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:226)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:219)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
The GetRealPathFromConn exception gets through about 20 times or so one right after the other. In the event Viewer I have of course 5 Warnings right before IIS shut down the pool:
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '3732'. The process exit code was '0xff'.
A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3052'. The data field contains the error number.
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '7972'. The process exit code was '0xff'.
A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4736'. The data field contains the error number.
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '8016'. The process exit code was '0xff'.
Application pool 'domain.com' is being automatically disabled due to a series of failures in the process(es) serving that application pool.