Isolating source of large pinned object count

Question

While capturing Performance Monitoring metrics for one of our .NET processes we noticed that we had a large number of pinned objects. Specifically, we were monitoring the ".NET CLR Memory" counter "# of Pinned Objects" which steadily rose in value into the thousands. In general the trend is a steady 45 degree upward trend line. Since this is a reliably repeatable condition we took a memory dump with WinDbg and were surprised to find that we only had 23 pinned items which did not match what we saw in PerfMon. This led us to the following questions:

1. Why is the PerfMon and GCHandles count of pinned objects different?

2. If PerfMon is correct doesn't this indicate a memory leak? If yes, how can we locate it?

Results from !gchandles

Note: all the "Pinned" handles are all System.Object[] which all appear to contain Int64 values

Handles:
    Strong Handles:       183
    Pinned Handles:       23
    Async Pinned Handles: 3
    Ref Count Handles:    7
    Weak Long Handles:    16762
    Weak Short Handles:   481
    SizedRef Handles:     8
    Dependent Handles:    57

Result from !eeheap -gc

0:048> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (0000000001fe2e20)
generation 0 starts at 0x0000000103879b20
generation 1 starts at 0x000000010385f528
generation 2 starts at 0x00000000ff801000
ephemeral segment allocation context: none
 segment     begin allocated  size
00000000ff800000  00000000ff801000  00000001038cfb38  0x40ceb38(67955512)
Large object heap starts at 0x00000004ff801000
 segment     begin allocated  size
00000004ff800000  00000004ff801000  0000000501006c98  0x1805c98(25189528)
Heap Size:               Size: 0x58d47d0 (93145040) bytes.
------------------------------
Heap 1 (0000000001fe7d50)
generation 0 starts at 0x0000000203d84328
generation 1 starts at 0x0000000203c58a70
generation 2 starts at 0x00000001ff801000
ephemeral segment allocation context: none
 segment     begin allocated  size
00000001ff800000  00000001ff801000  0000000204c9a298  0x5499298(88707736)
Large object heap starts at 0x000000050f801000
 segment     begin allocated  size
000000050f800000  000000050f801000  0000000510408e38  0xc07e38(12615224)
Heap Size:               Size: 0x60a10d0 (101322960) bytes.
------------------------------
Heap 2 (0000000001ff9590)
generation 0 starts at 0x000000030360fb10
generation 1 starts at 0x00000003036065f0
generation 2 starts at 0x00000002ff801000
ephemeral segment allocation context: none
 segment     begin allocated  size
00000002ff800000  00000002ff801000  00000003042ff190  0x4afe190(78635408)
Large object heap starts at 0x000000051f801000
 segment     begin allocated  size
000000051f800000  000000051f801000  0000000520c38850  0x1437850(21198928)
Heap Size:               Size: 0x5f359e0 (99834336) bytes.
------------------------------
Heap 3 (00000000020152b0)
generation 0 starts at 0x00000004036f9da8
generation 1 starts at 0x00000004036f9a28
generation 2 starts at 0x00000003ff801000
ephemeral segment allocation context: none
 segment     begin allocated  size
00000003ff800000  00000003ff801000  00000004038c3da0  0x40c2da0(67906976)
Large object heap starts at 0x000000052f801000
 segment     begin allocated  size
000000052f800000  000000052f801000  00000005305a1d88  0xda0d88(14290312)
Heap Size:               Size: 0x4e63b28 (82197288) bytes.
------------------------------
GC Heap Size:            Size: 0x1670eda8 (376499624) bytes.

Answer 1

That perf counter doesn't actually display the number of pinning handles. It displays the number of objects that could not be moved during the last garbage collection.

Which is a very different number. For one it is stale of course so your minidump is not necessarily a good match. For another, it also shows pins that don't require a handle. You can't easily see those back in the Windbg report. They are local variables that have the [pinned] attribute. The garbage collector finds them back when it performs a stack walk, a very efficient way to pin. The fixed keyword in C# creates them.

Seeing those kinds of pins grow has a limited number of possible causes that I can think of. Recursion can explain it. Or an ever increasing number of threads, a common cause for memory explosion trouble.