Apache/Tomcat error - wrong pages being delivered

Question

This error has been driving me nuts. We have a server running Apache and Tomcat, serving multiple different sites. Normally the server runs fine, but sometimes an error happens where people are served the wrong page - the page that somebody else requested!

Clues:

• The pages being delivered are those that another user requested recently, and are otherwise delivered correctly. It's been known for two simultaneous requests to be swapped. As far as I can tell, none of the pages being incorrectly delivered are older than a few minutes.
• It only affects the files that are being served by Tomcat. Static files like images are unaffected.
• It doesn't happen all the time. When it does happen, it happens for everybody.
• It seems to happen at times of peak demand. However, the demand is not yet very high - it's certainly well within the bounds of what Apache can cope with.
• Restarting Tomcat fixed it, but only for a few minutes. Restarting Apache fixed it, but only for a few minutes.
• The server is running Apache 2 and Tomcat 6, using a Java 6 VM on Gentoo. The connection is with AJP13, and JkMount directives within <VirtualHost> blocks are correct.
• There's nothing of use in any of the log files.

---

Further information:

Apache does not have any form of caching turned on. All the caching-related entries in httpd.conf and related imports say, for example:

<IfDefine CACHE>
  LoadModule cache_module modules/mod_cache.so
</IfDefine>

While the options for Apache don't include that flag:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D LANGUAGE -D SSL -D SSL_DEFAULT_VHOST -D PHP5 -D JK"

Tomcat likewise has no caching options switched on, that I can find.

toolkit's suggestion was good, but not appropriate in this case. What leads me to believe that the error can't be within my own code is that it isn't simply a few values that are being transferred - it's the entire request, including the URL, parameters, session cookies, the whole thing. People are getting pages back saying "You are logged in as John", when they clearly aren't.

---

Update:

Based on suggestions from several people, I'm going to add the following HTTP headers to Tomcat-served pages to disable all forms of caching:

Cache-Control: no-store
Vary: *

Hopefully these headers will be respected not just by Apache, but also by any other caches or proxies that may be in the way. Unfortunately I have no way of deliberately reproducing this error, so I'm just going to have to wait and see if it turns up again.

I notice that the following headers are being included - could they be related in any way?

Connection: Keep-Alive
Keep-Alive: timeout=5, max=66

---

Update:

Apparently this happened again while I was asleep, but has stopped happening now I'm awake to see it. Again, there's nothing useful in the logs that I can see, so I have no clues to what was actually happening or how to prevent it.

Is there any extra information I can put in Apache or Tomcat's logs to make this easier to diagnose?

---

Update:

Since this has happened again a couple of times, we've changed how Apache connects to Tomcat to see if it affects things. We were using mod_jk with a directive like this:

JkMount /portal ajp13

We've switched now to using mod_proxy_ajp, like so:

ProxyPass /portal ajp://localhost:8009/portal

We'll see if it makes any difference. This error was always annoyingly unpredictable, so we can never definitively say if it's worked or not.

---

Update:

We just got the error briefly on a site that was left using mod_jk, while a sister site on the same server using mod_proxy_ajp didn't show the error. This doesn't prove anything, but it does provide evidence that swithing to mod_proxy_ajp may have helped.

---

Update:

We just got the error again last night on a site using mod_proxy_ajp, so clearly that hasn't solved it - mod_jk wasn't the source of the problem. I'm going to try the anonymous suggestion of turning off persistent connections:

KeepAlive Off

If that fails as well, I'm going to be desperate enough to start investigating GlassFish.

---

Update:

Dammit! The problem just came back. I hadn't seen it in a while, so I was starting to think we'd finally sorted it. I hate heisenbugs.

Answer 1

I had this problem and it really drove me nuts. I dont know why, but I solved it turning off the Keep Alive on the http.conf

from

KeepAlive On

to

KeepAlive Off

My application doesn't use the keepalive feature, so it worked very well for me.

Answer 2

Could it be the thread-safety of your servlets?

Do your servlets store any information in instance members.

For example, something as simple as the following may cause thread-related issues:

public class MyServlet ... {
    private String action;

    public void doGet(...) {
         action = request.getParameter("action");
         processAction(response);
    }

    public void processAction(...) {
         if (action.equals("foo")) {
             // send foo page
         } else if (action.equals("bar")) {
             // send bar page
         }
     }
}

Because the serlvet is accessed by multiple threads, there is no guarantee that the action instance member will not be clobbered by someone elses request, and end up sending the wrong page back.

The simple solution to this issue is to use local variables insead of instance members:

public class MyServlet ... {
    public void doGet(...) {
         String action = request.getParameter("action");
         processAction(action, response);
    }

    public void processAction(...) {
         if (action.equals("foo")) {
             // send foo page
         } else if (action.equals("bar")) {
             // send bar page
         }
     }
}

Note: this extends to JavaServer Pages too, if you were dispatching to them for your views?

Answer 3

Although you did mention mod_cache was not enabled in your setup, for others who may have encountered the same issue with mod_cache enabled (even on static contents), the solution is to make sure the following directive is enabled on the Set-Cookie HTTP header:

CacheIgnoreHeaders Set-Cookie

The reason being mod_cache will cache the Set-Cookie header that may get served to other users. This would then leak session ID from the user who last filled the cache to another.

Answer 4

Are you sure that is the page that somebody else requested or a page without parameters?, you could get weird errors if your connectionTimeout is too short at server.xml on the tomcat server behind apache, increase it to a bigger number:

default configuration:

  <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

changed:

  <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="2000000"
               redirectPort="8443" />

Answer 5

Try this:

response.setHeader("Cache-Control", "no-cache"); //HTTP 1.1
response.setHeader("Pragma", "no-cache"); //HTTP 1.0
response.setDateHeader("Expires", 0); //prevents caching at the proxy server