Java Spring application has memory leak. system no

Question

I have monitor My Web application using yourkit profiler. There is major object that retain maximum size is SessionFactoryImpl, webappclassloader, and CGlib object Shows. *Does spring crone scheduler cause memory leak? solution That i tried

1) i tried to kill thread but still they are alive.

2) closed all connection.

3) allocate null to all variable and objects which i used in my code.

4) I also applied serverside

-Xms128m -Xmx256m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:MaxHeapFreeRatio=70 -XX:ReservedCodeCacheSize=32m -XX:+UseCodeCacheFlushing -XX:-OmitStackTraceInFastThrow

5) i have add leak prevention library in web.xml

<listener>
    <listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener
        </listener-class>
  </listener>
  <context-param>
    <param-name>ClassLoaderLeakPreventor.stopThreads</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <param-name>ClassLoaderLeakPreventor.stopTimerThreads</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <param-name>ClassLoaderLeakPreventor.executeShutdownHooks</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <param-name>ClassLoaderLeakPreventor.threadWaitMs</param-name>
    <param-value>5000</param-value>
  </context-param>
  <context-param>
    <param-name>ClassLoaderLeakPreventor.shutdownHookWaitMs</param-name>
    <param-value>10000</param-value>
  </context-param>
  <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/root-context.xml</param-value>
  </context-param> 

6) i have also add ContextFinalizer class

 package com.thl.test;

import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.sql.Driver;
import java.sql.DriverManager;
import java.util.Enumeration;
import java.util.Set;

import javax.servlet.ServletContextEvent;

import org.apache.commons.logging.LogFactory;
import org.springframework.beans.CachedIntrospectionResults;
import org.springframework.cglib.proxy.Enhancer;
import org.springframework.cglib.proxy.Proxy;
import org.springframework.context.ApplicationContext;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.web.util.IntrospectorCleanupListener;

import com.mysql.jdbc.AbandonedConnectionCleanupThread;

import se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor;

public class ContextFinalizer extends IntrospectorCleanupListener {

    private ClassLoader loader = null;

    public void contextInitialized(ServletContextEvent sce) {
        System.out.println("Calling>>>>>>>>>>>>>>>>>>>>>>>.?");
        /* Introspector.flushCaches(); */
        ClassLoader cl1 = Thread.currentThread().getContextClassLoader();
        CachedIntrospectionResults.clearClassLoader(cl1);
        LogFactory.releaseAll();
        ClassLoaderLeakPreventor.gc();
        try {
            AbandonedConnectionCleanupThread.shutdown();
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        /*Enhancer.registerCallbacks(enhanced, null);*/
        // cleanUp();
    }

    @SuppressWarnings("deprecation")
    public void contextDestroyed(ServletContextEvent sce) {
        /*
         * Thread t = Thread.currentThread();
         * Runtime.getRuntime().addShutdownHook(t);
         */
        System.out.println("Good Bye>>>>>>>>>>>>>>>>>>>>>.?");
        cleanUp();
        ClassLoaderLeakPreventor.gc();
        java.beans.Introspector.flushCaches();
        java.security.Security.removeProvider(null);
        ClassLoader cl1 = Thread.currentThread().getContextClassLoader();
        CachedIntrospectionResults.clearClassLoader(cl1);
        LogFactory.releaseAll();
        org.apache.log4j.LogManager.shutdown();

        Enumeration<Driver> drivers = DriverManager.getDrivers();
        Driver d = null;

        ClassLoader cl = Thread.currentThread().getContextClassLoader();

        while (drivers.hasMoreElements()) {
            try {
                d = drivers.nextElement();
                if (d.getClass().getClassLoader() == cl) {
                    DriverManager.deregisterDriver(d);
                } else {
                    DriverManager.deregisterDriver(d);
                }
            } catch (Exception ex) {
                // LOGGER.warn(String.format("Error deregistering driver %s",
                // d), ex);
            }
        }

        /*
         * if (ConnectionImpl.class.getClassLoader() ==
         * getClass().getClassLoader()) { Field f = null; try { f =
         * ConnectionImpl.class.getDeclaredField("cancelTimer");
         * f.setAccessible(true); Timer timer = (Timer) f.get(null);
         * timer.cancel(); }catch(Exception e) {
         * 
         * }finally { f = null; } }
         */

        try {
            com.mysql.jdbc.AbandonedConnectionCleanupThread.shutdown();

        } catch (InterruptedException e) {
        } finally {
            try {

                /* org.apache.commons.pool.impl.GenericObjectPool. */

                com.mysql.jdbc.AbandonedConnectionCleanupThread.shutdown();
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

        Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
        Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
        for (Thread t : threadArray) {
            /*
             * if (t.isInterrupted()) { break; }
             */

            if (t.getName().contains("Abandoned connection cleanup thread")) {
                synchronized (t) {
                    // don't complain, it works
                    if (t.isAlive()) {
                        System.out.println("Alive True");
                        if (t.isDaemon()) {
                            System.out.println("isDaemon True");
                            t.stop();
                        } else {
                            System.out.println("isDaemon False");
                            t.stop();
                        }
                    } else {
                        System.out.println("Alive Flase");
                        t.stop();
                    }
                    // new Timer(true);
                }
            } else if (t.getName().contains("http-nio-8081-exec-1")) {
                System.out.println("http-nio-8081-exec-1>>>>>>>>>>>");
            } else {
                System.out.println("Else If Block");
                synchronized (t) {
                    t.setDaemon(true);
                    t.suspend();
                }
            }
        }
        java.beans.Introspector.flushCaches();

    }

    public void onApplicationEvent(ContextRefreshedEvent arg0) {
        System.out.println("--------------- Context Refreshed -----------------");
        System.out.println("::::::::::::::::::::::::  Calling   :::::::::::::::::::::::::::::");

        ApplicationContext context = arg0.getApplicationContext();
        System.out.println(context.getDisplayName());
    }

    private void cleanUp() {
        Thread[] threads = getThreads();
        for (Thread thread : threads) {
            if (thread != null) {
                System.out.println("Inside IFF");
                cleanContextClassLoader(thread);
                cleanOrb(thread);
                cleanThreadLocal(thread);

            }

        }
    }

    private Thread[] getThreads() {
        ThreadGroup rootGroup = Thread.currentThread().getThreadGroup();
        ThreadGroup parentGroup;
        if (rootGroup.getParent() != null) {
            parentGroup = rootGroup.getParent();
            if (parentGroup != null) {
                rootGroup = parentGroup;
            }
        }
        Thread[] threads = new Thread[rootGroup.activeCount()];
        while (rootGroup.enumerate(threads, true) == threads.length) {
            threads = new Thread[threads.length * 2];
        }
        return threads;
    }

    private boolean loaderRemovable(ClassLoader cl) {
        if (cl == null) {
            return false;
        }
        Object isDoneCalled = getObject(cl, "doneCalled");
        String clName = cl.getClass().getName();
        loader = Thread.currentThread().getContextClassLoader();
        String ldr = null;
        loader = loader.getParent();
        if (loader != null) {
            // loader.getParent();
            ldr = loader.getClass().getName();
        }

        if (clName != null && ldr != null && isDoneCalled != null) {
            if (clName.equalsIgnoreCase(ldr) && isDoneCalled instanceof Boolean && (Boolean) isDoneCalled) {
                return true;
            }
        }

        return loader == cl;
    }

    private Field getField(Class clazz, String fieldName) {
        Field f = null;
        try {
            f = clazz.getDeclaredField(fieldName);
        } catch (NoSuchFieldException ex) {

        } catch (SecurityException ex) {
        }

        if (f == null) {
            Class parent = clazz.getSuperclass();
            if (parent != null) {
                f = getField(parent, fieldName);
            }
        }
        if (f != null) {
            f.setAccessible(true);
        }
        return f;
    }

    private Object getObject(Object instance, String fieldName) {
        Class clazz = instance.getClass();
        Field f = getField(clazz, fieldName);
        if (f != null) {
            try {
                return f.get(instance);
            } catch (IllegalArgumentException | IllegalAccessException ex) {
            }
        }
        return null;
    }

    private void cleanContextClassLoader(Thread thread) {
        if (loaderRemovable(thread.getContextClassLoader())) {
            thread.setContextClassLoader(null);
        }
    }

    private void cleanOrb(Thread thread) {
        Object currentWork = getObject(thread, "currentWork");
        if (currentWork != null) {
            Object orb = getObject(currentWork, "orb");
            if (orb != null) {
                Object transportManager = getObject(orb, "transportManager");
                if (transportManager != null) {
                    Thread selector = (Thread) getObject(transportManager, "selector");
                    if (selector != null && loaderRemovable(selector.getContextClassLoader())) {
                        selector.setContextClassLoader(null);
                    }
                }
            }
        }
    }

    private void removeThreadLocal(Object entry, Object threadLocals, Thread thread) {
        ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent");
        if (threadLocal != null) {
            Class clazz = null;
            try {
                clazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
            } catch (ClassNotFoundException ex) {
            }
            if (clazz != null) {
                Method removeMethod = null;
                Method[] methods = clazz.getDeclaredMethods();
                if (methods != null) {
                    for (Method method : methods) {
                        if (method.getName().equals("remove")) {
                            removeMethod = method;
                            removeMethod.setAccessible(true);
                            break;
                        }
                    }
                }
                if (removeMethod != null) {
                    try {
                        removeMethod.invoke(threadLocals, threadLocal);
                    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                    }
                }

            }

        }
    }

    private void cleanThreadLocal(Thread thread) {
        Object threadLocals = getObject(thread, "threadLocals");
        if (threadLocals != null) {
            Object table = getObject(threadLocals, "table");
            if (table != null) {
                int size = Array.getLength(table);
                for (int i = 0; i < size; i++) {
                    Object entry = Array.get(table, i);
                    if (entry != null) {
                        Field valueField = getField(entry.getClass(), "value");
                        if (valueField != null) {
                            try {
                                Object value = valueField.get(entry);
                                if (value != null && value instanceof ClassLoader
                                        && loaderRemovable((ClassLoader) value)) {
                                    removeThreadLocal(entry, threadLocals, thread);
                                }
                            } catch (IllegalArgumentException | IllegalAccessException ex) {

                            }

                        }
                    }

                }
            }
        }
    }

}

snapshot about memory leak monitor is given below.

enter link description here

Answer 1

there is an answer here

What you are seeing there is a symptom, not the cause of your leak. What cause the leak is that the ClassLoader is never collected, in which case the CachedIntrospectionResults are of little effect in comparaison to the memory taken by every classes and their static members. If your ClassLoader is not collected, you might even have external resources not freed, such as JDBC Connections, if you have a reference from a static member to your connection pool (directly or indirectly).

Any servlet container will release the ClassLoader when the webapp is unloaded, but there's many things (bugs) that might prevent the ClassLoader to be collected. The most probable causes I was personally able to identify are the cache in java.lang.Introspector, a Driver left unregistered in DriverManager or ThreadLocal variable not cleared. But in any case, unless there was a bug in the patch I provided, or that it somehow got broken since (in both cases I doubt it), then code in CacheIntrospectionResults will never prevent the ClassLoader to be garbage collected.

So, most likely, it's not a JBoss problem, nor a Spring problem, but a problem in your own code or in libraries such as DOM4J.

For the Introspector, the solution (beside not using it), is to have a ContextListener that call Introspector.flushCaches() when the context is destroyed. Spring provide one, it's os.web.util.IntrospectorCleanupContextListener or something like that. Also, the leak in the Introspector is only an issue for JDK up to JDK 1.4.2, the problem was corrected in 1.5.

For the a Driver that wasn't unregistered (if your driver class is in WEB-INF/lib), the solution is similar, you need to have a ContextListener, when the context is destroyed, you ask for a list of the registered Drivers from DriverManager, and you remove any that comes from your webapp (you can check its ClassLoader).

For the ThreadLocal stuff, you have to make sure anything you put in there you eventually remove it. But if a third party libary does that (e.g. DOM4J) can you can't fix it or have it fixed, then it's more complicated to solve. You can look in a previous email from me to this list for a pontential workaround, but it's kind of a hack, and would be dangerous in production because of concurrency issues

.

Answer 2

If you think your spring code has some problem, please filter your package on jvisualvm then you can see your memory of instance.

In my opinion, I think you use and config some other library not correctly, it may take some memory problem on you application. Let check the most related pakages on the jvisualvm, It make helpful for you