CLR out Of Memory Exceptions

问题:

We are getting a lot of out of memory exceptions and we cant seem to diagnose what is happening. It seems to be an issue that happens that will spike memory usage from 300 megs to over a Gig in a mater of a few Min. Now this is an IIS app and has 3 app domains running in separate thread pools.

We want to record when the Memory exceptions are about to happen. So we can attempt to find a pattern. My question is: What is the best way to do this? Is there a way to Query the Memory usage once a min to see how high it is and send an alert email. Or maybe write an app that will monitor the CLR's memory usage? an ideas or a direction are more then welcome.

EDIT

I am using Perfmon but unless I am watching the process it is not that useful. I can only see what is was and when. I have also used the Red Gate Memory Profile tool, which is awesome by the way, only I cannot seem to hit apon the page or process that is causing the exception.

回答1:

Install ADPlus on your server (it's part of the Windows Debugging Tools). When you start observing unusually high memory usage capture a memory dump using:

adplus -hang -p <PID> -quiet -o <dump file folder path>

<PID> is the process ID of the worker process which you can get from tasklist.exe.

If you're not always around when this issue occurs then you could automate capturing a memory dump using DebugDiag:

1. Use DebugDiag in leak tracking mode to trigger a process dump when either your Private or Virtual memory use reaches a certain threshold. That said, I've not always found this reliable.

2. Use DebugDiag in Crash Mode to capture a dump whenever CLR exceptions are thrown. You can used the Advanced Settings to configure DebugDiag to produce a full memory dump upon encountering a CLR exception of type System.OutOfMemoryException. This is more reliable and will definitely trigger. Only use Advanced Settings -> Exceptions, don't touch the Unconfigured First Chance Exceptions, leave this setting at None.

Once you have your memory dump fire up WinDBG, load the dump file and load up SOS and start poking about.

Tess Ferrandez's blog is a great .NET debugging resource and she has plenty of articles and labs about how to track down memory leaks:

If it is broken, fix it you should - memory issue articles

.NET Debugging Demos Lab 6:_Memory Leak
.NET Debugging Demos Lab 6:_Memory Leak - Review
.NET Debugging Demos Lab 7: Memory Leak
.NET Debugging Demos Lab 7: Memory Leak - Review

PerfMon counters are useful and can be used to confirm that you have a memory leak, but they don't tell the whole story. WinDBG and SOS are the tools you need to use to find out where your memory is being used.

回答2:

Perfromance monitor aka Perfmon is your friend - It is free and not intrusive and light weight and can be safely run on production servers if you use less frequent sampling (every few seconds). The minimum it can do is to sample memory/cpu usage for your processes (w3wp.exe) and store them on a file.

Since you have not shared what you are doing in the application I cant suggest performance counters to store but there are plenty in the ASP.NET and .NET and CLR.

Since you are getting CLR out of memory exception, my hunch is GC is not working due to a pinned object or something. I doubt it is unmanaged resource such as bitmap not being released although it could be.

Here are a list of counters I suggest:

.NET CLR Memory

• GC Handles
• Pinned Objects

Process for the w3wp.exe running your app

• A couple but mostly Working Set

ASP.NET

• Managed Memory used

回答3:

Try RedGate Memory Profiler. I suppose it works with ASP.NET sites (their performance profiler definitely does).

回答4:

Use a memory profiler. There are a couple of good ones out there, e.g. JetBrains dotTrace or ANTS Memory Profiler from Red Gate. There were couple of discussions here at stackoverflow with lots of other tips and recommendations.

回答5:

If you cannot afford a good profiler, you should utilize what Microsoft recommends from here, http://msdn.microsoft.com/en-us/library/ee817663.aspx

回答6:

I would use ADPlus in "crash" mode to capture a memory dump when the exception occurs, and then WinDbg and SOS to figure out what's taking up all of the memory.

回答7:

Use dotTrace or YourToolkit .Net where you can attach the profiler to the ASP process, they have trial version so you don't need to spend money right away. Using these profilers you can select the time line where memory starts increasing (you can see the memory usage visually in a graph) so it should be quite easy to select the range and understand what is causing the memory usage to shoot up so high.