how to set max heap size for Python

问题:

I'd like to set max heap size for my python code. In my server environment, a process in running long time with huge heap size is killed automatically. I already know if heap size is above 40G, the process will be killed. And if around 20G~30G, it's OK. Thus, I need to keep max heap size within 20G.

To realize that, I found similar question in stackoverflow(How to limit the heap size?), and tried it.

here is header of my code.

import resource
rsrc = resource.RLIMIT_DATA
resource.setrlimit(rsrc, (7340032, 20971520))

But my code exceeds that limit if I check process status with top command. It takes around 45G memory.

What's wrong with my way to set limit?

My python code executes PCA analysis with scikit-learn library. I confirmed numpy array size is around 6G with nbytes attribute.

thanks!