I have code that looks similar to this:
def memoryIntensiveFunction(x):
largeTempVariable = Intermediate(x)
processFunction(largeTempVariable,x)
The problem is that the variable temp
is something like 500 mb in a test case of mine, but that space is not returned to the OS when memoryIntensiveFunction
is finished. I know this because memory profiling with the guppy
tool says largeTempVariable
is freed (i.e., within Python), but psutil
shows it isn't. I presume I'm seeing the effects described here. The problem is that this process is long running (i.e. hours), memoryIntensiveFunction
is run at the beginning and never again, so it's inconvenient for me to have to carry the 500mb around for hours.
One solution I found here and here suggests using a separate process. Multiprocessing incurs its own costs, but it would be worth it in my case. However, this would require refactoring memoryIntensiveFunction
callers to receive x
as a return value instead of seeing it modified in place. The real killer is that my object x
is not picklable (it makes heavy use of boost python extensions). It would be a lot of work to make x
picklable.
Are there any options I'm not considering?
This seem curious enough that I tried to reproduce your issue, and seems that simple "del" was plenty. To demonstrate, you can run the following code:
And at first breakpoint you will see that it uses roughly 1gb of ram (you want the python3.3 entry):
And then at second breakpoint, after we del the variable the memory is freed:
Now if we drop the "del" from function, and set a breakpoint right after test():
The memory indeed won't be freed before we terminate:
So my suggestion? Just delete the sucker after you've used it, and do not need it any more ;)