I have to load this massive object A
(that can weight almsot 10go) that I need to pass to a function, that extracts from it a parameter B
to further exert some heavy computations on it.
A = load(file)
def function(A):
B = transorm(A)
B = compute(B)
return(B)
In order to free some memory (as I already had a MemoryError), I'd like to remove A
from memory right after its transformation to B.
I tried del
but it doesn't seem to affect the existence of A
at the script level. I also tried del global()["A"]
but it says that A is not defined as a global variable.
Is there a way to do it? Thanks!
I believe reassigning A within the function could achieve the effect you are looking for.
Perhaps loading the object from inside the function would work here, since
A
will go out of scope once the function returns and no longer take up the memory in the same way(A
will probably still exist in memory, but that memory should now be available for other use again when needed). Maybe try something like this:Then just call
function_B(file)
del A
will simply removeA
from the local scope offunction
(see this answer).A
will still persist in the global scope. To remove it from the global scope you can either use a closure (and declareglobal A
) or withpython3
you can also use the keywordnonlocal
. However this only removes the binding from the scope and does not guarantee that the corresponding memory is freed. This happens when the object is garbage collected. You can force garbage collection via thegc
module (see this answer).However if you are running into memory problems, instead of loading the whole data set, you could maybe use a view onto the data set and only process (load) a part of it at a time (i.e. stream-process the data).