We have a VB6 app that calls out to a .NET DLL. Occasionally, after the VB6 app has been running for a long time and has called the .NET code a lot, the .NET side of things throws an OutOfMemory exception, even though there is plenty of memory available on the machine. The VB6 memory space is also no where near it's limit.
Does the .NET side keep a separate memory pool? Or is it apart of the VB6 app's memory pool?
If it is separate, is there a way to see how big it is? The only huge memory items in my Task Manager are SQL Server and the VB6 app (both expected).
This doesn't happen too often, but when it does, it's hard to pin down why the system won't allocate more memory.
It seems like a memory leak somewhere, and assuming the dll and calling app are correct, it may be in the call. Check the parameter data types, and byref vs. byval. Parameters in .net default to byval, in vb6 byref. There are various string types in each that are not always converted properly on a call to a library.
The answer ended up being very simple:
A .NET DLL built with DEBUG configuration will leak while running.
Switching to a RELEASE build fixed my issue.
Background:
I finally got ANTS to debug the VB6 app and see the .NET process (had to change the VB6 code to load the .NET code as soon as possible). Once I did that, I saw a huge number of weak referenced objects whose parent was __ENCList. This classes allows for Edit and Continue during debugging. A quick Google search showed immediately this was caused by using a DEBUG build.
My Google Search
Links:
WeakReferences in Debug Build
First thing to check is the pinning of objects in your memory. In a multi-threaded environment this can get out of control fairly quickly depending on how the code is written. When .NET goes to grab more memory it will take it in 8, 16, or 32 MB blocks and those blocks need to be completely clean. That is, you may have hundreds of MB of free memory but if there isn't a 32 MB block free without anything else in it then you get the OOM you've been seeing. I highly suggest getting a memory profiler such as ANTS and taking a closer look at things.
More often than not this error should be read as out of GDI Objects. Check the GDI/Handles counter in Task Manager processes tab or use GDIView.
