Symptoms: ASP.NET Web Forms website on Azure sporadically crashes and all .aspx page requests fail with this error. The problem seems random and only happens once in a great while. The site may run for months with no issues then out of the blue it will stop serving any .aspx pages and gives the error on every .aspx page request. A restart of the website is the only solution (or redeploy, which causes the same thing).

This was a very difficult problem to debug since it was so sporadic and none of the other answers I found helped, they did not address the problem where the site would deploy and run for long periods of time then crash with this error seemingly randomly. In the end I got some help from Microsoft.

According to Microsoft this is caused by a known bug with the .Net Framework versions 4.7.x triggered by memory pressure in the web server worker process. It is fixed in .Net framework 4.8 (not yet released at this time). My solution was simply to check "allowed precompiled site to be updatable" and check the "Do not merge" option.

Details from Microsoft:

The root cause of the issue is relating to the memory pressure on the worker process (w3wp.exe) address space. In ASP.net, when the .Net Framework detects that we have exceeded a certain memory pressure threshold, it will proceed to try and eject items from the internal caching structures in order to relieve space. Upon such a cache trim, assemblies belonging to your application are removed from the memory cache. This triggers a callback delegate that will try and reflect that changes in memory cache onto the cache of assemblies on disk – and will try and delete the .dll assembly ejected from the in-memory cache. However, the worker process is still keeping a reference to this file (an open handle) and as such, the delete fails. When this occurs, the .Net Framework creates a .delete file next to the assembly to mark it as stale. This will prevent the worker process from loading the file back into memory cache and causes the compilation error you are seeing.

The cause of all of this cache processing is an incorrect insertion into the ASP.net memory cache of dynamic assemblies issued from the pre-compilation process with non-updatable UI. In this process of compilation, the names of the dynamic assemblies cannot be changed, as the .compiled resource files which indicate to the ASP.net Runtime where a compiled resource is located (in what binary) do not change if changes are made to the site – contrary to the pre-compilation with updatable UI, where the markup for pages can be subject to change, and this generates changes to the .compiled files and then to the names of the dynamic assemblies, thus preventing the re-usage of the old names.

For assemblies that have come from the pre-compilation of an ASP.net application with non-updatable UI, the standard was to insert them into the ASP.net memory cache in the worker process with a special attribute that would indicate to the cache manager that these entries cannot be removed. This attribute is missing in the .Net Framework 4.7.x release builds and is the cause of the error. Without it, when the cache is trimmed, the assemblies can be removed, and since they are still in usage, they cannot be deleted from the Temporary ASP.net Files folder (the shadow copy folder). As such, this results in the creation of the .delete files. This was not an issue in builds of the .Net Framework prior to 4.7. The product group has also confirmed that they have resolved this issue in the upcoming .Net Framework 4.8 release.

There are a couple of solutions around this issue:

Remove the 4.7.x .Net Framework and re-install the 4.6.x Framework.

The caching error is only present in builds of the .Net Framework of 4.7 and above, thereby reverting to a 4.6.x version would allow you to continue working without facing the problem. They can remain in the 4.6.x distribution until the .Net Framework 4.8 is released later this year. I do not have a timeline from the product group for this release as of now.

Increase RAM and private bytes recycling limitations. (Does not apply to Azure webapps)

The cache is trimmed in ASP.net when we see that the memory pressure is too great inside the worker process. This memory pressure is evaluated as follows: if you do not have a recycling condition set in the application’s application pool in IIS, ASP.net will consider the threshold to be 60 % of the total RAM available to the machine. When the private bytes of the worker process exceed this threshold, the internal cache will be trimmed, and the assemblies will be ejected, causing the appearance of the .delete files. I advise you work with the customer to increase the RAM memory available to their most impacted servers if possible, and also set a memory recycling based on private bytes on the app pool hosting the impacted application. We can set this recycling limitation to a higher value than the RAM so that we can ensure the threshold is high enough not to be reached by the worker process.

To set the private bytes recycling on the application pool:

Start the IIS Manager console on the impacted servers.
Right click the impacted application pool and chose ‘Recycling’ from the context menu that is displayed.
In the recycling properties window that is displayed, introduce a value for the ‘Private Memory Usage (KB)’ textbox.
The value should be 1.2 * available RAM expressed in KB. This is a temporary workaround, and should be rolled back once the new version of the .Net Framework is released and installed.

Install the .Net Framework 4.8 Preview

You can try to install the preview version of the .Net Framework 4.8 which is available for download online now. However, this is a preview version and you may run into other issues which will not be supported until the Framework is released. You can also opt to install a private fix from Microsoft (reach out to me if you want this nand I will request it), which will change the malfunctioning assemblies from the .Net Framework 4.7.x to allow correct insertion of assemblies into the cache. However, this private fix is not signed, and will hence require us to disable assembly sign checking on the server which may expose the customer to other security risks.

In project precompile options, check "Allowed precompiled site to be updatable" and "Do not merge" options.

The bug only affects non-updatable precompiled binaries, so this avoids the bug, but startup times are obviously greatly affected. For me, this solution was simple and works fine until 4.8 rolls out for Azure web services.