Virtual and Physical Memory / OutOfMemoryException

问题:

I am working on a 64-bit .Net Windows Service application that essentially loads up a bunch of data for processing. While performing data volume testing, we were able to overwhelm the process and it threw an OutOfMemoryException (I do not have any performance statistics on the process when it failed.) I have a hard time believing that the process requested a chunk of memory that would have exceeded the allowable address space for the process since its running on a 64-bit machine. I do know that the process is running on a machine that is consistently in the neighborhood of 80%-90% physical memory usage. My question is: Can the CLR throw an OutOfMemoryException if the machine is critically low on available physical memory even though the process wouldn't exceed it's allowable amount of virtual memory?

Thanks for your help!

回答1:

There are still some reachable limits in place in a 64-bit environment. Check this page for some of the most common ones. In short, yes, you can still run out of memory, if your program loads a whopping 128GB of data into virtual memory. You could also still be limited by the 2GB max per-process limit if you do not have the IMAGE_FILE_LARGE_ADDRESS_AWARE environment variable set.

回答2:

Another possibility is that the program tried to allocate a single block of memory larger than 2 gigabytes, which is a .NET limitation. This can happen when adding things to a collection (most often a Dictionary or HashSet, but also a List or any other collection that grows automatically.)

Dictionary and HashSet do this often if you're trying to put more than about 47 million items into the collection. Although the collection can hold about 89.5 million, the algorithm that grows the collection does so by doubling. If you start from an empty Dictionary and begin adding items, the collection doubles a number of times until it reaches about 47 million. Then it tries to double again and throws OutOfMemoryException.

The way to avoid the exception is to pre-allocate the collection so that its capacity is large enough to hold however many items you expect to put into it.

回答3:

There's what you can theoretically address.

There's what physical me mory you have, and whe nyou exceed that you start using swap, which is often limited to the size of some chosen disk partitions.

As a rule of thumb one tends to have a small number (like one or two) of multiples of physical memory as swap.

So yes it's quite likely that you are out of available, as opposed to addressable memory.