Memory leak testing and constant variables

Question

I'm working on tests to detect memory leaks for classes. I do this with the following pattern:

• Get initial memory in test setup (using GC.GetTotalMemory(true))
• Perform operations, such as instantiation of objects
• Dispose object (through useage block)
• Get memory in test tear down (using the same method again, all objects are out of scope and therefore should be garbage collected)
• Ideally, assert that the different between initial and end memory = 0

Using memory profiling tools in my tests, I see that const string (and related) declarations are counted towards the leaked memory for a test. I'd like these values to be already declared when doing the initial memory measurement, such that they do not generate a 'leak' that is detected by the test. However, I don't want to run the action code twice, because that would prevent me from detecting leakage of variables in static variables (like not having cleaned up singleton pattern object that should not have application life-time).

Is there a way to do this? Do I have to do a manual compensation for these variables per test case (which has the drawback of having to be actively maintained, which will probably won't happen in the long run)? Is there a moment in runtime which I can use for this purpose? Or will I have to just accept that I cannot test for a memory leak near 0 bytes?

(For those interested: I'm already using WeakReferences and livelyness checks to monitor if objects that I instantiate in the test are properly garbage collected, but this won't cover private state that leaks)

Answer 1

It's more or less futile trying to find small memory leaks in managed memory. Instead, if possible, try a much bigger scope - allocate the objects 100 000 times, and see if there's a significant difference on cleanup.

In practice, you probably want to do something completely different, though. Amount of memory is generally not a very reliable check for managed leaks. Instead, you might want to let your application run over and over in a cycle, and use e.g. the CLRProfiler to see the lifetime of objects. If there's something suspicious that hangs on for minutes or even hours, that's probably a leak. This gives you a much better idea than just checking memory size.

Of course, in general, you shouldn't really care. There's very little benefit in hunting a memory leak until it actually hurts you. Do note that I'm talking only about managed memory. Any place where you're not using managed memory only should be given significant attention - whether it's native code, unsafe pointers or any other form of an unmanaged handle. Do note that this includes networking code (sockets have unmanaged memory), file operations (file handles), some synchronization primitives (wait handles), image data (most notably GDI+'s Bitmap class, WPF is full of these as well) and more. Generally, though, using high-level approaches, you can usually safely ignore most issues until they actually noticeably manifest under realistic conditions.

Now, all that said, your methodology is suspect anyway. You have to force the GC to run (using the current implementation, it will only collect based on memory pressure, not because you left scope - and stack is quite a separate issue as far as GCing is concerned). The usual practice to force a full GC is like this:

1. GC.Collect();
2. GC.WaitForPendingFinalizers();
3. GC.Collect();

The whole process then needs you to first warm everything up, ie. do one complete cycle of the test first. Then you do a full forced GC as noted above. Then you do (say) a thousand cycles of the test. And after that, you do another full GC. This should give you a much clearer picture of real memory usage.

Static field memory leaks are only interesting if they're either working with a lot of memory (in which case, you should simply ask why the hell is this using so much memory, and what can I do about it?), or if they increase over multiple calls or over time (in which case, this methodology will still let you discover the issue).

Also, what do you mean, a singleton object that does not have the scope of the application? If that's what you need, you really, really shouldn't be using a singleton! You have very little control over when the singleton is created or destroyed, and you can't recreate it after it has been destroyed. And of course, most people will probably tell you not to use singletons at all nowadays - one of the reasons being the fact that unless they have no state, they more or less kill any chance of using practices such as unit testing.