Long running application slows down

Question

There is an application that consists of three executable files. One of them - a dispatcher, which runs other executables. The dispatcher receives a code from an executable at its completion. That is, only the dispatcher is always running, other executables unloads and loads again. The application runs on the point of service and work around the clock. At the first launch the application works fast. At the end of the day, the application works terribly slow. What could be the reason for such a behavior?

Answer 1

There could be lot of reasons for a slow down over time. Anywhere from a slow memory leak to anti-virus. The best you can do is try to build evidence (data) about what area of the application to look first. Try not to talk it over with many devs because everyone will have a different opinion about what might be wrong. Get the data!

How to get the data:

perfmon perfmon is your friend. There are a lot of counters that you can look at (system wide as well as process specific). So you can start by profiling the big 4 (that's memory, disk usage, cpu and networking). There are a lot of posts out there about what counters are best, so I won't go into too much detail about the perf counters here.

windbg If you indeed see that memory is growing and not being collected it's time to bring in the big guns. .NET is great at abstracting memory usage away from developers, but this means we have to get underneath .NET sometimes to find out what is not allowing the Garbage Collector to do its work. windbg with the sos.dll (managed extensions) is a great tool for this. The hardest part of windbg (in my experience) is just getting the sos extensions loaded properly. You have to pay close attention to what target architecture (64 or 32) you are analysing and what CLR version you are running on.

procdump procdump by sysinternals is a great little utility to take memory snapshots from a running process. These snapshots (.dmp files) can then be analyzed by windbg.

sos The sos.dll has shipped with the .NET Framework since v2. With v4, Visual Studio 2010 has integrated sos and allows you to analyze .dmp files!

The sos commands for memory leaks that I have found most useful are:

!eeheap -gc (overview of what is in each generation of each heap)

!dumpheap -min <size> (dumps out all objects and types, over a particular <size>)

!dumpheap -type <type> (dump out all objects of a specific <type>)

!gcroot <address> (prints out a stack so you can see what parent object is pinning in the GC)

!do <address> (prints out memory of a specific object)

Some other pointers:

Usually, you want to snapshot memory under load, so it would be good to have some way to simulate that from outside the system. So, it is good to get this running ahead of time and even work it into the QA process for the application.

For performance problems it is usually best to take regular snapshots over time with a running application. Then you can compare the snapshots when you analyse.

Well, that was a bit longer than I intended, but hopefully worth it!

Answer 2

You have to check the memory usage of your dispatcher application... it seems you are not disposing of unused objects.