Automated way to find JUnit tests that leak memory

Question

The root of our problem is Singletons. But Singletons are hard to break and in the meantime we have a lot of unit tests that use Singletons without being careful to completely clear them in the tearDown() method. I figure that a good way to detect tests like these is to look for memory leaks. If the memory used after tearDown() and System.gc() is more than when the test started, either the test leaked or more classes were loaded by the classloader. Is there any way to automatically detect this sort of problem?

Answer 1

I completely agree with other posters that monitoring the memory usage isn't a viable way to track this - System.gc() is not going to behave as you expect, or with enough precision to achieve your goal.

You're going to need a tool that lets you inspect the reference graph and show allocation call stacks.

I've used OptimizeIt from Borland and JProfiler from ej-technologies, both with success (a quick google reveals that OptimizeIt may now be dead.)

There's also the possiblity of using JVMTI to throw together a better monitor for this specific problem.

Edit: Wierd, but as I was reviewing this answer, I got a phone call from Embarcadero, who has apparently purchased OptimizeIt, done some updating and are now marketing under the name J Optimizer.

Answer 2

If your Singleton's are only intended to be initialized one time, you could have code that checks for reinitialization and logs the current stack when it detects that. Then if you check the stack, you will see which test got the ball rolling, and you can check the JUnit logs to see what the test run right before that was.

In terms of solving this problem more thoroughly, instead of detecting it I would recommend you have a singleton initializer that remembers what it initialized, and has one teardown method that tears down everything it initialized. That way tests can be made to only initialize via this class, and only has to do one thing in teardown.

I also think Carl Manaster's suggestion is a good one, but if you were using JUnit4, then you could have a teardown method that runs in the superclass without having to remember to call super. Unless you use the JUnit3 GUI, JUnit4 should be a drop in replacement. The only thing is to take advantage of its new features you have to migrate the whole test, you can't have both live in the same class. So tests that interact with these singletons would have to be migrated one whole test class at a time.

Answer 3

Just a thought: if you have two empty tests run right after one another, the second one should not have a different memory used after teardown(). If it does, you (probably) have a leak somewhere in your setup()/teardown() system.

Answer 4

You could use the Eclipse Memory Analyzer to automate analyzing heap dumps taken after each test or probably better after all tests. MAT can find memory leaks fairly automatically.

Answer 5

Could you introduce a subclass, between TestCase and your individual test classes, that did the cleanup? Then subclasses would only be responsible for calling super.teardown() - and only those that had a teardown() of their own.

Answer 6

I don't think this is a good approach. System.gc() is not guaranteed to fully clean up any unused objects as you think it will.

If your root problem is that you have unit tests which end up using global data (singletons) without properly cleaning them up, you should attack the root problem: these unit tests. It shouldn't be too too hard to find all tests that aren't using tearDown(), or to find all tests that use a particular singleton.