ASP.NET Web application doesn't unload AppDoma

Question

When we deploy our web application we copy all the code to a new directory and then point iis to that new directory. When we do this the number of appdomains increases but never decreases. Also, our Application_End event never seems to fire.

For some unknown time, while both sets of AppDomains are still reported by perfmon, the system performs very poorly while % time in GC spikes to 100%. Eventually I recycle the app pool to get the app running smoothly again.

Extra piece of info: listing the appdomains shows 2 on my dev machine, but 4 run on the live server... we just have one application running in the pool so this means some library we're using is creating app domains.

What should I do to try to debug what's happening? What would prevent an app domain from unloading?

Update 9/3/2014

After getting some more detail log information it looks like the problem is not old app domains staying around, it's new appdomains created during restart. Instead of the application starting one new instance, it starts two. Sometimes we get application_end from the old instance, sometimes we don't.

Update 9/4/2014

Both things are happening. Using process explorer I can see on one of the machines that the old app domain is still there and a new one has started. On the other machine there were only 2 app domains but there was a gap in their sequential ids. So two instances started (we also get a log message from app start), one of them died off almost instantly, leaving 2 app domains.

Answer 1

One thing you could check, to help troubleshoot:

Process Explorer actually has a .NET Assemblies tab which lists all the AppDomains loaded by a process. NOTE: The tab only appears for processes that use the .NET Framework.

Answer 2

Although I am not sure this is the ONLY reason the app was not unloading appdomains, it certainly is one reason. The actual answer is much less interesting than the steps I used to figure it out.

1. I wrote a little ruby script to just make 100s of requests in 10 threads against our app ( setting up jmeter would take more time than this).
2. Run script and made changes to the app config file many times while it ran. This took a couple of minutes.
3. I used process explorer to confirm that there were indeed 4 app domains still loaded into the process.
4. I ran procdump and created a dump file.
5. Loaded the procdump file into visual studio and clicked "Debug with Managed Only" (even though I had no idea if the bug was in managed code or not)
6. Debugger stopped on a line of code where the thread that called Application_End was waiting for a queue to finish processing. By looking at the values of the variables I was able to tell that the queue was no longer going to process items but we were going to wait until the queue was empty.
7. Changed the code and start process over again, this time all the app domains started by my changes to the web config were unloaded.