Memory Overflow: Having an increasing number of Mi

Question

We are currently hunting some memory leaks in our application, when doing some operation(loading and closing one project inside our application), we know that the memory increase always a little bit.

We have already found a lot of them, but now, the 10+ most increasing classes are (according to our tool, ANTS Memory Profiler 8.2):

• Microsoft.CSharp.RuntimeBinder.Semantics.SYMTBL+Key
• Microsoft.CSharp.RuntimeBinder.Semantics.LocalVariableSymbol
• Microsoft.CSharp.RuntimeBinder.Semantics.CONSTVAL
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRCONSTANT
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRCLASS
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRTYPEOF
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRLIST
• Microsoft.CSharp.RuntimeBinder.Semantics.MethWithInst
• Microsoft.CSharp.RuntimeBinder.Semantics.CMemberLookupResults
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRMEMGRP
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRCALL
• Microsoft.CSharp.RuntimeBinder.Semantics.EXPRWRAP
• Microsoft.CSharp.RuntimeBinder.Semantics.AggregateDeclaration
• Microsoft.CSharp.RuntimeBinder.Semantics.Scope

Unfortunately, I don't what this is, so It's a little bit hard for me to find how/what I should release.

I checked the instance tree but, it goes all the way with microsoft stuff.

The issue is that when we do the "Open/close" of a project, we go through a lot(most of) our code.

EDIT One part of our application uses the dynamic keyword for some resources, it may be linked. The class here are not Disposable, should I do something special with them?

EDIT 2

I'm pretty sure this is related to my dynamic stuff, it seems that C# create a cache when using dynamic. But currently I've no idea why it grows(I load the same classes all the time, and I will have exactly the same signature all the time), nor how to clear this.

Answer 1

I encountered exactly the same issue today by profiling memory leaks in my app RepoZ. That tool is supposed to run in the background, checking Git repositories and update Windows Explorer window titles periodically. The latter task has to make some COM calls to "Shell.Application" to find the Explorer windows and determine the path they are currently pointing to.

By using the dynamic keyword like this ...

dynamic shell = Activator.CreateInstance(...);
foreach (object window in shell.Windows())
{ 
    var hwnd = window.Hwnd;
    ...
}

... I ended up to a memory dump like that after a few hours:

Combridge

To solve that, I wrote a little helper class called "Combridge" caring to release COM objects and providing quite easy access to methods and properties of the underlying COM object. It is very easy and straightforward, nothing special here. It makes use of Reflection to COM objects, that's why there's some loss in performance (see below).

With it, the code sample from above looks like this:

using (var shell = new Combridge(Activator.CreateInstance(...)))
{
    var windows = shell.InvokeMethod<IEnumerable>("Windows");
    foreach (var window in windows)
    {
        var hwnd = window.GetPropertyValue<long>("Hwnd");
        ... 
    }
}

You can see the file ExplorerWindowActor on how it is used in RepoZ.

It is not exactly as beautiful as with dynamic and performance got worse in this first attempt as well. A quick bench showed the following:

Performance

I tested 1000 iterations, in each iteration 10 open Explorer windows were processed. For each window, 4 methods or properties are invoked on that COM object. So we're talking about 40.000 COM calls.

The duration went up from ~2500ms (dynamic) to ~6000ms (Combridge). That's from 0.062ms to 0.150ms for each call.

So this takes about 2.4x the time to finish now.

This is significant, I know. But it is okay for my requirements and the memory leak is gone.

That's it - I wanted to share that story with you, hopefully you can use that class (or an improved version of it) to get out of the dynamic hell as well.

~Update~

After 10 hours, RepoZ still runs with a very constant memory footprint.

So with 10 Explorer windows open, 4 COM calls per window and that whole loop two times a second, RepoZ created about 72.000 COM instances and made about 2.880.000 COM calls in total without any increase in memory consumption.

I guess we can say that the issue really comes with dynamic.

Answer 2

Dynamic keyword should be used seldom since in most of the cases workarounds can be found not requiring it.

Based on your application, the best advise is to carefully think if you can design your solution so that you avoid dynamic. Here are some valid use cases for dynamic: https://msdn.microsoft.com/en-us/library/dd264736.aspx

Given that you really need to use dynamic, I would suggest instrumenting your code and figure out what parts are the most memory consuming. Indeed using dynamic increases your memory consumption based on the fact that it needs to perform all kinds of lookups, but to have a out-of-memory exception you would need to use a lot of dynamic variables for a lot of unknown types.

There are a lot of different ways for calling methods on unknown types, and measuring and tuning the bottlenecks is the way to go.

PS: Also, posting some code snippets helps a lot.