Locating code that is filling PermGen with dead Gr

We have had our glassfish instance go down every two weeks for a while with a java.lang.OutOfMemoryError: PermGen space. I increased the PermGen space to 512MB and startet dumping memory usage with jstat -gc. After two weeks I came up with the following graph that shows how the PermGen space is steadily increasing (the units on the x-axis are minutes, y-axis are KB).

I tried googling around for some kind of profiling tool that could pinpoint the error and a thread here on SO mentioned jmap, which proved to be quite helpful. Out of the approximately 14000 lines dumped from jmap -permstats $PID, approximately 12500 contained groovy/lang/GroovyClassLoader$InnerLoader, pointing to some kind of memory leak from either our own Groovy code or Groovy itself. I have to point out that Groovy constitues less than 1% of the relevant codebase .

Example output below:

class_loader    classes bytes   parent_loader   alive?  type

<bootstrap> 3811    14830264      null      live    <internal>
0x00007f3aa7e19d20  20  164168  0x00007f3a9607f010  dead    groovy/lang/GroovyClassLoader$InnerLoader@0x00007f3a7afb4120
0x00007f3aa7c850d0  20  164168  0x00007f3a9607f010  dead    groovy/lang/GroovyClassLoader$InnerLoader@0x00007f3a7afb4120
0x00007f3aa5d15128  21  181072  0x00007f3a9607f010  dead    groovy/lang/GroovyClassLoader$InnerLoader@0x00007f3a7afb4120
0x00007f3aad0b40e8  36  189816  0x00007f3a9d31fbf8  dead    org/apache/jasper/servlet/JasperLoader@0x00007f3a7d0caf00
....

So how can I proceed to find out more about what code is causing this?

From this article I infer that our Groovy code is dynamically creating classes somewhere. And from the the dump from jmap I can see that most of the dead objects/classes(?) have the same parent_loader, although I am unsure what that means in this context. I do not know how to proceed from here.

Addendum

For latecomers, it's worth pointing out that the accepted answer does not fix the issue. It simply extends the period needed before rebooting with a tenfold by not storing so much class info. What actually fixed our problems was getting rid of the code that generated it. We used the validation (Design by contract) framework OVal where one could script custom constraints using Groovy as annotations on methods and classes. Removing the annotations in favor of explicit pre- and post-conditions in plain Java was boring, but it got the job done. I suspect each time an OVal constraint was being checked a new anonymous class was being created and somehow the associated class data was causing a memory leak.

回答1:

We had a similar problem (1 week between crashes). The trouble seems to be that Groovy caches meta methods. We ended up using this code based on this discussion and bug report

GroovyClassLoader loader = new GroovyClassLoader();
Reader reader = new BufferedReader(clob.getCharacterStream());
GroovyCodeSource source = new GroovyCodeSource(reader, name, "xb3.Classifier");
Class<?> groovyClass = loader.parseClass(source);
Object possibleClass = groovyClass.newInstance();
if (expectedType.isAssignableFrom(possibleClass.getClass())) {
    classifiers.put((T) possibleClass, name);
}
reader.close();
// Tell Groovy we don't need any meta
// information about these classes
GroovySystem.getMetaClassRegistry().removeMetaClass(possibleClass.getClass());
// Tell the loader to clear out it's cache,
// this ensures the classes will be GC'd
loader.clearCache();

回答2:

If you are using Sun JVM change it for IBM JVM , it will works fine i hope :)