Procdump Error writing dump file: 0x80070005 Error

Question

Recently we have had issues when trying to capture a memory dump for various IIS application pools on a 2012 R2 server. I've tried using task manager, but it generates an error, as well as using procdump in an Administrative console:

PS C:\Users\_______\Downloads> procdump -mA 31016

ProcDump v7.1 - Writes process dump files
Copyright (C) 2009-2014 Mark Russinovich
Sysinternals - www.sysinternals.com
With contributions from Andrew Richards

[19:59:22] Dump 1 initiated: C:\Users\____\Downloads\w3wp.exe_161008_195922.dmp
[19:59:25] Dump 1 writing: Estimated dump file size is 29278 MB.
[20:01:15] Dump 1 error: Error writing dump file: 0x80070005
Error 0x80070005 (-2147024891): Access is denied.

[20:01:15] Waiting for dump to complete...
[20:01:17] Dump count not reached.

I have tried various combinations of -ma and -mA and -64 and procdump64 but they all have the same Access is denied error for worker processes using over ~16GB of memory.

I also tried adding -r to reflect/clone the process as recommended by How to: Take a Memory Dump of an ASP.NET Application Pool Quickly but still get the same error message as above.

Update: So by default IIS application pools will be recycled if they do not respond to ongoing internal ping requests within 90 seconds. You can see this in the advanced settings:

And the error message occurs after about 90 seconds, so likely that is the cause of the issue.

Answer 1

So what is happening is procdump suspends the worker process which prevents it from responding to the internal pings, even when using the -r reflect/clone option. And if writing the memory to the dump file takes longer than 90 seconds then IIS will recycle the worker, causing the old process to be terminated. Procdump then returns an "Access Denied" or "Only part of a ReadProcessMemory or WriteProcessMemory request was completed" error message, because the memory it was trying to read is no longer allocated and the process no longer exists.

To work around this issue you can use Resouce Monitor, Process Explorer or PsSuspend to also suspend the svchost.exe -k iissvcs process so that it cannot interrupt the procdump process. The following PowerShell script can be run in an Admin console to create a memory dump of the w3wp process with the largest working set:

#Prevent IIS from recycling the process during procdump and causing an Access Denied error message
$iispid = Get-Process svchost | ?{$_.modules.ModuleName -eq "iisw3adm.dll"} | Select -First 1 -ExpandProperty Id
$workerpid = Get-Process w3wp | Sort ws -Descending | Select -First 1 -ExpandProperty Id
cd ~\Downloads  #move to location where you want to save the dump files
#Add -accepteula to the sysinternals calls if you want to bypass the initial EULA prompt on new servers
& "c:\sysinternals\pssuspend.exe" $iispid 
Write-Output "Creating memory dump for w3wp PID $workerpid"
& "c:\sysinternals\procdump.exe" -ma $workerpid
& "c:\sysinternals\pssuspend.exe" $iispid -r

The output should look something like this:

PS> & "\\dfshare\sysinternals\pssuspend.exe" $iispid

PsSuspend v1.06 - Process Suspender
Copyright ⌐ 2001-2003 Mark Russinovich
Sysinternals

Process 49836 suspended.

PS> & "\\dfshare\sysinternals\procdump.exe" -ma 98340

ProcDump v8.2 - Sysinternals process dump utility
Copyright (C) 2009-2016 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

[01:03:24] Dump 1 initiated: C:\Users\gbray\Downloads\w3wp.exe_161230_010324.dmp
[01:03:29] Dump 1 writing: Estimated dump file size is 19347 MB.
[01:05:14] Dump 1 complete: 19350 MB written in 109.8 seconds
[01:05:14] Dump count reached.

PS> & "\\dfshare\sysinternals\pssuspend.exe" $iispid -r

PsSuspend v1.06 - Process Suspender
Copyright ⌐ 2001-2003 Mark Russinovich
Sysinternals

Process 49836 resumed.

I have no idea what other issues suspending the iissvcs process might create, so it may be best to run iisreset after the memory dump is created.