I have a .NET windows service application with a memory leak.
Overview: The service connects to another service via TCP, and writes data to a SQL server database. The service normally runs between 100 and 300 MB depending on how much data it has queued up. It generally takes 3 + days before I noticed the service's memory is getting out of hand, and will have a Out Of Memory exception within 2 weeks of running.
I think the issue might be related to a massive number of objects that are not leaving the finalization queue.
I have attached to the service with Windbg and run the !FinalizationQueue command and get:
0:029> !finalizequeue
SyncBlocks to be cleaned up: 0
Free-Threaded Interfaces to be released: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
generation 0 has 0 finalizable objects (32c69ea4->32c69ea4)
generation 1 has 266 finalizable objects (32c69a7c->32c69ea4)
generation 2 has 23191 finalizable objects (32c53020->32c69a7c)
Ready for finalization 5124616 objects (32c69ea4->33ff66c4)
Statistics for all finalizable objects (including all objects ready for finalization):
MT Count TotalSize Class Name
73e0b3ec 1 16 System.Threading.Gen2GcCallback
73e16648 1 20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle
73e165f8 1 20 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle
73df36f4 1 20 System.Security.Cryptography.SafeKeyHandle
...
73e2b208 36 720 Microsoft.Win32.SafeHandles.SafeWaitHandle
731a7754 33 924 System.Net.SafeFreeCredential_SECURITY
731a2d58 10 1240 System.Diagnostics.PerformanceCounter
59d01074 28 1680 System.Management.ManagementObject
73e2a8c4 175 2100 System.WeakReference
70b0385c 48 6144 System.Data.SqlClient.SqlConnection
731b2e94 201 6432 System.ComponentModel.Container
70b03ef4 86 14104 System.Data.SqlClient.SqlCommand
70b06014 712 14240 System.Data.SqlClient.SNIPacket
73e15b34 1248 24960 System.Threading.ThreadPoolWorkQueueThreadLocals
73e26c74 2238 44760 Microsoft.Win32.SafeHandles.SafeRegistryHandle
**73e2e424 1903 98956 System.Threading.Thread
70b08530 986 295800 System.Data.DataTable
70b08b08 20802 2662656 System.Data.DataColumn
6fab76dc 379139 7582780 System.Transactions.SafeIUnknown
73e30008 4740054 208562376 System.Threading.ReaderWriterLock**
I ran: !DumpHeap on the ReaderWriterLock and got:
0:029> !DumpHeap /d -mt 73e30008
... Omitted
Address MT Size
00c51f58 73e30008 44
00c5644c 73e30008 44
00c5ebc0 73e30008 44
00c81420 73e30008 44
... Omitted
I was trying to find out why there are millions of ReaderWriterLocks in the finalization queue, so I ran GCRoot on several of the addresses from the dumpheap and get:
0:029> !GCRoot 00c5ebc0
HandleTable:
005413cc (pinned handle)
-> 01d133c8 System.Object[]
-> 00cd2820 System.Diagnostics.TraceSource
-> 00cd29ac System.Diagnostics.SourceSwitch
-> 00c71704 System.Diagnostics.SwitchElementsCollection
-> 00c7177c System.Configuration.ConfigurationValues
-> 00c5fbec System.Configuration.RuntimeConfigurationRecord
-> 00c5ebec System.Configuration.RuntimeConfigurationRecord
-> 00c5eb74 System.Configuration.Internal.InternalConfigRoot
-> **00c5ebc0** System.Threading.ReaderWriterLock
Found 1 unique roots (run '!GCRoot -all' to see all roots).
I ran !do on the lock address:
0:029> !do 00c5ebc0
Name: System.Threading.ReaderWriterLock
MethodTable: 73e30008
EEClass: 73a6ff54
Size: 44(0x2c) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
73e2c00c 4001890 4 System.IntPtr 1 instance 0 _hWriterEvent
73e2c00c 4001891 8 System.IntPtr 1 instance 0 _hReaderEvent
73e2c00c 4001892 c System.IntPtr 1 instance 0 _hObjectHandle
73e2f6bc 4001893 10 System.Int32 1 instance 0 _dwState
73e2f6bc 4001894 14 System.Int32 1 instance 0 _dwULockID
73e2f6bc 4001895 18 System.Int32 1 instance 3 _dwLLockID
73e2f6bc 4001896 1c System.Int32 1 instance 0 _dwWriterID
73e2f6bc 4001897 20 System.Int32 1 instance 3 _dwWriterSeqNum
73e2c494 4001898 24 System.Int16 1 instance 0 _wWriterLevel
I wanted to see all the modules that had the ReaderWriterLock loaded, so I ran the Name2EE command on the System.Threading.ReaderWriterLock and got:
0:029> !Name2EE * System.Threading.ReaderWriterLock
Module: 739f1000
Assembly: mscorlib.dll
Token: 020004c8
MethodTable: 73e30008
EEClass: 73a6ff54
Name: System.Threading.ReaderWriterLock
Module: 00573fbc
Assembly: J_ToDatabase.exe
Module: 00574e0c
Assembly: JLibrary.dll
Module: 72f31000
Assembly: System.ServiceProcess.dll
Module: 72f71000
Assembly: System.dll
Module: 00578034
Assembly: JConfigHelper.dll
Module: 72141000
Assembly: System.Windows.Forms.dll
Module: 72da1000
Assembly: System.Drawing.dll
Module: 0057ac64
Assembly: JLib.dll
Module: 0057ba1c
Assembly: J_DB_Deploy.dll
Module: 0057c8a0
Assembly: JModel.dll
Module: 03571d7c
Assembly: JLicense.dll
Module: 71211000
Assembly: System.Xml.dll
Module: 03574074
Assembly: JSecureBase.dll
Module: 03574804
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 71921000
Assembly: System.Configuration.dll
Module: 71a11000
Assembly: System.Core.dll
Module: 0357539c
Assembly:
Module: 03575c7c
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 59ce1000
Assembly: System.Management.dll
Module: 03577b2c
Assembly: Localization.dll
Module: 03578b6c
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03579070
Assembly:
Module: 03579840
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 709f1000
Assembly: System.Data.dll
Module: 702f1000
Assembly: System.Security.dll
Module: 70661000
Assembly: System.Numerics.dll
Module: 6faa1000
Assembly: System.Transactions.dll
Module: 6f4e1000
Assembly: System.EnterpriseServices.dll
Module: 6f471000
Assembly: System.EnterpriseServices.Wrapper.dll
Module: 043a2cbc
Assembly: JTPlib.dll
Module: 043a3af8
Assembly: JStore.Core.dll
Module: 043a4920
Assembly: SystemEventArgsUtility.dll
Module: 043a7378
Assembly: JMAFLibrary.dll
Module: 043ace28
Assembly: J_A_Library.dll
Module: 03c0afb8
Assembly: DatabaseSharedLibrary.dll
Module: 03ab1c24
Assembly:
Module: 03ab23f8
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03ab28f8
Assembly:
Module: 03ab30c8
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03ab35c8
Assembly:
...
**NOTE JSecure showed up like this over 800 times!**
...
Module: 03ab3d98
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Assembly: JSecure, Version=17.13.21.82, Culture=neutral This seems to be part of the problem because it showed up over 800 times, I omitted all instances from the results because it was too much text.
Notes about JSecure, it is a dll that manages the licensing of my service. The dll is loaded into my service via
System.Reflection.Assembly.Load
method because of security reasons. The two major questions I have are
- What is causing JSecure at least seem to be loaded over 800 times?
- What is causing millions of System.Threading.ReaderWriterLock to be left in the finalization queue?
Any clues on how to track down this memory leak further?
Update 1
Per suggestion in the comment, I have dug into the finalizer thread more. When I run !Threads, I get 1905 results, though I think most of the threads are also in the finalizerqueue, there are only about 20 active threads. When I change to the Finalizer thread, and run kb to get the stack trace, I get:
0:002> kb
# ChildEBP RetAddr Args to Child
00 02e2f0e0 75892cc7 000003d4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01 02e2f154 75892c02 000003d4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x99
02 02e2f168 75ff9839 000003d4 ffffffff 0075d59c KERNELBASE!WaitForSingleObject+0x12
03 (Inline) -------- -------- -------- -------- combase!MTAThreadWaitForCall+0x43 [d:\blue\com\combase\dcomrem\channelb.cxx @ 5657]
04 02e2f198 7609d524 006e1568 006e5808 02e2f404 combase!MTAThreadDispatchCrossApartmentCall+0x1ed [d:\blue\com\combase\dcomrem\chancont.cxx @ 193]
05 (Inline) -------- -------- -------- -------- combase!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x33a2 [d:\blue\com\combase\dcomrem\channelb.cxx @ 5052]
06 02e2f2ec 75f7caea 0075d59c 02e2f404 02e2f3cc combase!CRpcChannelBuffer::SendReceive2+0x62d [d:\blue\com\combase\dcomrem\channelb.cxx @ 4796]
07 (Inline) -------- -------- -------- -------- combase!ClientCallRetryContext::SendReceiveWithRetry+0x2e [d:\blue\com\combase\dcomrem\callctrl.cxx @ 1090]
08 (Inline) -------- -------- -------- -------- combase!CAptRpcChnl::SendReceiveInRetryContext+0x81 [d:\blue\com\combase\dcomrem\callctrl.cxx @ 715]
09 02e2f34c 75fc1789 0075d59c 02e2f404 02e2f3cc combase!DefaultSendReceive+0x9e [d:\blue\com\combase\dcomrem\callctrl.cxx @ 671]
0a (Inline) -------- -------- -------- -------- combase!CAptRpcChnl::SendReceive+0x38 [d:\blue\com\combase\dcomrem\callctrl.cxx @ 752]
0b 02e2f3b0 7609a010 0075d59c 02e2f404 02e2f3cc combase!CCtxComChnl::SendReceive+0x248 [d:\blue\com\combase\dcomrem\ctxchnl.cxx @ 735]
0c 02e2f3d4 77865769 0073be14 02e2f430 77865740 combase!NdrExtpProxySendReceive+0x5c [d:\blue\com\combase\ndr\ndrole\proxy.cxx @ 2017]
0d 02e2f3ec 778e6c1b ee46341f 0073be14 02e2f850 rpcrt4!NdrpProxySendReceive+0x29
0e 02e2f818 76099e1e 75f6d8f8 75f72ff0 02e2f850 rpcrt4!NdrClientCall2+0x22b
0f 02e2f838 75f7c46f 02e2f850 00000008 02e2f8d8 combase!ObjectStublessClient+0x6c [d:\blue\com\combase\ndr\ndrole\i386\stblsclt.cxx @ 215]
10 02e2f848 75fc1c07 0073be14 02e2f888 006ce0a8 combase!ObjectStubless+0xf [d:\blue\com\combase\ndr\ndrole\i386\stubless.asm @ 171]
11 02e2f8d8 75fc3024 006ce098 74d08aa9 02e2f99c combase!CObjectContext::InternalContextCallback+0x1e4 [d:\blue\com\combase\dcomrem\context.cxx @ 4428]
12 02e2f92c 74d086b1 006ce0a8 74d08aa9 02e2f99c combase!CObjectContext::ContextCallback+0xbc [d:\blue\com\combase\dcomrem\context.cxx @ 4332]
13 02e2fa24 74d08782 74d081d7 02e2fab4 ec1c2444 clr!CtxEntry::EnterContext+0x243
14 02e2fa5c 74d087c3 74d081d7 02e2fab4 00000000 clr!RCW::EnterContext+0x3a
15 02e2fa78 74d085a0 006ce150 ec1c2490 752963e8 clr!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xc0
16 02e2fad4 74cd19c5 ec1c253c 7529562c 00000000 clr!RCWCleanupList::CleanupAllWrappers+0x14d
17 02e2fb24 74cd1bc8 006ae690 00000001 ec1c24e0 clr!SyncBlockCache::CleanupSyncBlocks+0xd0
18 02e2fb3c 74cd3ad8 ec1c2550 02e2fc58 02e2fcd0 clr!Thread::DoExtraWorkForFinalizer+0x7f
19 02e2fb84 74d09b0e 02e2fcd0 006bdcc8 02e2fcd0 clr!WKS::GCHeap::FinalizerThreadWorker+0x232
1a 02e2fb98 74d09b78 ec1c2224 02e2fcd0 00000000 clr!ManagedThreadBase_DispatchInner+0x67
1b 02e2fc3c 74d09c45 ec1c2280 00000000 00000040 clr!ManagedThreadBase_DispatchMiddle+0x7e
1c 02e2fc98 74c852e2 ec1c230c 74c8517c 00000000 clr!ManagedThreadBase_DispatchOuter+0x5b
1d 02e2fd14 74cd5f91 00000000 00000000 00000000 clr!WKS::GCHeap::FinalizerThreadStart+0x1a0
1e 02e2fdb4 77917c04 006bd910 77917be0 ee68e49d clr!Thread::intermediateThreadProc+0x4d
1f 02e2fdc8 77cdab8f 006bd910 ee10b078 00000000 kernel32!BaseThreadInitThunk+0x24
20 02e2fe10 77cdab5a ffffffff 77cbffdf 00000000 ntdll!__RtlUserThreadStart+0x2f
21 02e2fe20 00000000 74cd5f48 006bd910 00000000 ntdll!_RtlUserThreadStart+0x1b
{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://img.manongdao.com/sparkler-677774__340.jpg', '推荐 我命由我不由天 的文章《Memory Leak, why are there so many ReaderWriterLoc》','https://www.manongdao.com/article-577284.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}