Tomcat 7 stops receiving HTTP-requests

Question

I have a Tomcat 7 server that receives a lot of GET-requests. This works very well for some time, then suddenly it stops working (after 7-8 hours).

When it stops working, I am getting this error:

mai 06, 2015 12:47:58 AM org.apache.coyote.http11.AbstractHttp11Processor process INFO: Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.

All others errors after this, is just the same. I have tried to stop the get-requests, but it crashes after a while anyways. Do you have any ideas?

This is right before it crashes:

[06/May/2015:10:47:11 +0200] "GET /ListenerServlet?request=getLastChanged&_=1430852558104 HTTP/1.1" 200 32

After the crash:

[06/May/2015:10:47:14 +0200] "GET /ListenerServlet?request=getLastChanged&_=1430902035081 HTTP/1.1" 500 1674

After the crash, the Servlet can't get the request parameter.. that throws this nullpointer:

SEVERE: Servlet.service() for servlet [RoomServlet] in context with path [] threw exception java.lang.NullPointerException
        at factory.Factory.initEditMode(Factory.java:37)
        at factory.Factory.init(Factory.java:20)
        at factory.TabsFactory.<init>(TabsFactory.java:19)
        at servlet.RoomServlet.getTabs(RoomServlet.java:285)
        at servlet.RoomServlet.doGet(RoomServlet.java:46)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

followed by:

mai 07, 2015 9:58:00 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class websocket.drawboard.DrawboardContextListener
java.lang.NoClassDefFoundError: javax/websocket/Endpoint
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2944)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1208)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1688)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1569)
        at websocket.drawboard.DrawboardContextListener.contextDestroyed(DrawboardContextListener.java:32)
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5014)
        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5659)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javax.websocket.Endpoint
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1718)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1569)
        ... 17 more

IF there is anything more I need to add, let me know! I really appreciate.

EDIT: Today was a good day, I got a new error! This might have something with the JDBC....

The last packet successfully received from the server was 33 921 291 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago. No operations allowed after connection closed.

Answer 1

You can try installing Apache Tomcat v7.0.53 or higher.

Answer 2

I found out the solution to this almost by random.

There was nothing related to:

org.apache.coyote.http11.AbstractHttp11Processor process

but it was rather a JDBC problem.. the connection closed after 8 hours, and I didn't get any errors in the Servlet' for this.

The solution for me was to add some lines in my application's Context:

I added:

url="jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true"
dontTrackOpenResources="true"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="300000"

This is my full Context now:

<Context>
  <Resource name="jdbc/test" auth="Container" type="javax.sql.DataSource"
               maxActive="100" maxIdle="100" maxWait="30000"
               username="test" password="bestpassword" 
               driverClassName="com.mysql.jdbc.Driver"
               url="jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true"
               removeAbandoned="true" removeAbandonedTimeout="10"
               factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
               fairQueue="true"
               dontTrackOpenResources="true"
               testWhileIdle="true"
               timeBetweenEvictionRunsMillis="300000"
               />
</Context>

My server has not crashed in a week now, and I believe it wont crash again.

Answer 3

Maybe you have a file descriptor leak, which could cause failure in class loading, you can check how many file descriptor are open by using lsof.