How do I use `setrlimit` to limit memory usage? RL

I'm trying to use setrlimit to limit my memory usage on a Linux system, in order to stop my process from crashing the machine (my code was crashing nodes on a high performance cluster, because a bug led to memory consumption in excess of 100 GiB). I can't seem to find the correct resource to pass to setrlimit; I think it should be resident, which cannot be limited with setrlimit, but I am confused by resident, heap, stack. In the code below; if I uncomment only RLIMIT_AS, the code fails with MemoryError at numpy.ones(shape=(1000, 1000, 10), dtype="f8") even though that array should be only 80 MB. If I uncomment only RLIMIT_DATA, RLIMIT_RSS, or RLIMIT_STACK both arrays get allocated successfully, even though the total memory usage is 2 GB, or twice the desired maximum.

I would like to make make my program fail (no matter how) as soon as it tries to allocate too much RAM. Why do none of RLIMIT_DATA, RLIMIT_RSS, RLIMIT_STACK and RLIMIT_AS do what I mean, and what is the correct resource to pass to setrlimit?

$ cat mwe.py 
#!/usr/bin/env python3.5

import resource
import numpy

#rsrc = resource.RLIMIT_AS
#rsrc = resource.RLIMIT_DATA
#rsrc = resource.RLIMIT_RSS
#rsrc = resource.RLIMIT_STACK

soft, hard = resource.getrlimit(rsrc)
print("Limit starts as:", soft, hard)

resource.setrlimit(rsrc, (1e9, 1e9))

soft, hard = resource.getrlimit(rsrc)
print("Limit is now:", soft, hard)
print("Allocating 80 KB, should certainly work")
M1 = numpy.arange(100*100, dtype="u8")

print("Allocating 80 MB, should work")
M2 = numpy.arange(1000*1000*10, dtype="u8")

print("Allocating 2 GB, should fail")
M3 = numpy.arange(1000*1000*250, dtype="u8")

input("Still here…")

Output with the RLIMIT_AS line uncommented:

$ ./mwe.py 
Limit starts as: -1 -1
Limit is now: 1000000000 -1
Allocating 80 KB, should certainly work
Allocating 80 MB, should work
Traceback (most recent call last):
  File "./mwe.py", line 22, in <module>
    M2 = numpy.arange(1000*1000*10, dtype="u8")
MemoryError

Output when running with any of the other ones uncommented:

$ ./mwe.py 
Limit starts as: -1 -1
Limit is now: 1000000000 -1
Allocating 80 KB, should certainly work
Allocating 80 MB, should work
Allocating 2 GB, should fail
Still here…

At the final line, top reports that my process is using 379 GB VIRT, 2.0 GB RES.

System details:

$ uname -a
Linux host.somewhere.ac.uk 2.6.32-573.3.1.el6.x86_64 #1 SMP Mon Aug 10 09:44:54 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 (Santiago)

$ free -h
             total       used       free     shared    buffers     cached
Mem:          2.0T       1.9T        37G       1.6G       3.4G       1.8T
-/+ buffers/cache:        88G       1.9T 
Swap:         464G       4.8M       464G 

$ python3.5 --version
Python 3.5.0

$ python3.5 -c "import numpy; print(numpy.__version__)"
1.11.1

Alas I have no answer for your question. But I hope the following might help:

Your script works as expected on my system. Please share exact spec for yours, might be there is a known problem with Linux distro, kernel or even numpy...
You should be OK with RLIMIT_AS. As explained here this should limit the entire virtual memory used by the process. And virtual memory includes all: swap memory, shared libraries, code and data. More details here.

You may add the following function (adopted from this answer) to your script to check actual virtual memory usage at any point:

def peak_virtual_memory_mb():
    with open('/proc/self/status') as f:
        status = f.readlines()
        vmpeak = next(s for s in status if s.startswith("VmPeak:"))
        return vmpeak