Tomcat Session Eviction to Avoid OutOfMemoryError

Question

We are running a vendor-supplied webapp in Tomcat 5.5 using the StandardManager for sessions (in memory). As sessions can get quite large (20M+), running out of heap space is a serious concern. Users want to keep sessions around for a couple of hours if possible but would rather evict sessions than run out of heap space. It does not appear that the vendor properly implemented Serializable in session'ed objects, so switching to the persistent session manager implementation isn't an option.

Tomcat allows one to set a maxActiveSessions property that will restrict the overall number of sessions in the manager. However, when that limit is reached, no new sessions may be created until some existing sessions expire. We want to clobber the least recently used sessions first.

Ideally, we would like to expire some not recently used sessions when heap usage approaches the "Xmx" setting even if they are not old enough to expire unconditionally. A very old Tomcat developer mailing list thread suggested that this might allow a denial of service attack*, but, as this application is only available within the corporate network, we are unconcerned.

I thought about extending StandardManager to override processExpires() and clobber additional sessions if heap usage is greater than, say, 85% of max. However, this seems a bit problematic in practice. What if muuch of the heap is unreferenced, and the garbage collector would be able to collect a ton of objects (if it bothered to run) to reduce heap to 50% of max? I'd be expiring sessions unnecessarily. I guess we could mitigate this risk with some aggressive garbage collection settings. Also, how would I know how much memory I had saved by expiring a session? I would have to wait for a few GC cycles to know for sure. Perhaps I could take a conservative approach and remove at most N sessions per background process cycle until memory drops below an acceptable threshold. I could serialize the session as a way to estimate how much stuff would be GC'ed, but that relies on the vendor implementing Serializable and marking instance variables as transient appropriately.

Has anyone solved this problem? As a short-term fix, we are increasing heap size, but that band-aid has its drawbacks as well.

• They were referring to a public site where sessions would be created pre-login. Someone could cause many new sessions to be created and crowd out the ones actually in use.

Update: We really don't have much control over the system's architecture, and we specifically can't reduce session use. However, we can futz with the container as much as we'd like. Tomcat is the only open source servlet container that's vendor-supported, however.

Answer 1

Your options seem to be:

1. reduce the idle session timeout,
2. make sessions persistent (maybe only after the user has logged in),
3. reduce the memory used by each session object,
4. increase the memory for your Tomcat instance,
5. run multiple instances of your service, and put a load balancer in front of it/them.

From a technical standpoint 3 is the best solution ... if it works. The others all have a down-side.

Doing clever things with memory is only a band-aid. From the users' perspective, it makes your site's behaviour harder to understand. Besides, if your user base / traffic is trending upwards, you are only putting off the problem of finding a sustainable solution.

Answer 2

I'd advise to front multiple tomcat instances with apache and then use mod_jk to load balance between them.

You can do this without any real clustering so session sharing won't be an issue.

Mod_jk is rock solid and even offers a simple gui to mark instances as out of use etc.

This would also bring many other benefits in terms of resilience etc. I've personally used this setup on a very large scale public facing site and it worked a charm.

The ideal situation is to setup true session sharing but in some cases this is overkill.

See here:

http://tomcat.apache.org/connectors-doc/generic_howto/quick.html

Answer 3

I agree with Stephen C that it's your vendor you need to take up the problem with - as he said even switch vendors. One thing that no one has mentioned here (surprisingly) is that the vendor should also look at cleaning unused session objects.

It's very easy, unless you keep it in mind all the time, to let session size bloat out - to let an object's size bloat out unnoticed - then have large session attribs when we have a list of these objects - then never cleaning them out - if the app is large where lists of objects A, B, C, D, E, F, are displayed in certain sections of the app, but never cleaned out then that's a fundamental housekeeping problem that the vendor has not addressed.

e.g. is there a 'central screen' in the webapp where you navigate to other parts of the app? If so then the vendor should be cleaning up when entering this screen all the objects which would have been collected and stuffed into session on the screens that are accessible from the central page/screen/portal

Edit And use pagination, not load whole lists meeting criteria (except when pagination is part of the criteria)

I hope that comment helps you, and others.

Answer 4

Have you tried increasing the max heap size of the JVM?

The default, if not specified, is only 64mb - which I would say is on the small side for most intensive/full-blown web applications.

The best way to do this with Tomcat is to add the following to setenv.bat/.sh:

export CATALINA_OPTS=-Xmx128m

(substitute whatever value you want for 128 if you'd like larger than 128mb. Also change to correct syntax for Windows / your shell)

The startup and catalina shell scripts for Tomcat have built-in logic to invoke this file if it exists. This is the "best practice" way to specify any custom environment properties you need to set for your Tomcat install - putting the properties in this file is better than editing startup.sh or catalina.sh directly because this file can be portable between Tomcat installations/versions.

You might also be interested in this link: 6 Common Errors in Setting Java Heap Size (which also has a section at the end on How to set java heap size in Tomcat?).