Using memory_profiler to aid in project that is requiring freeing up some memory at various points. The development environment is OS X snow leopard.
The profiled memory, as shown below, is peaking at around 414.699 MiB
, but the Activity Monitor is showing the process peaking at nearly twice that (more than 900 MB
).
Line # Mem usage Increment Line Contents
================================================
24 20.441 MiB 0.000 MiB @profile
25 def do_work():
26 "Call each function in order"
27 20.445 MiB 0.004 MiB x = audio.AudioQuantumList()
28 137.098 MiB 116.652 MiB audiofile = make_objects("/Users/path/audio/Track01.mp3")
29 295.480 MiB 158.383 MiB audiofile2 = make_objects("/Users/path/audio/Track02.mp3")
30 414.699 MiB 119.219 MiB audiofile3 = make_objects("/Users/path/audio/Track03.mp3")
31 414.699 MiB 0.000 MiB x = add_to_list(audiofile, x)
32 417.426 MiB 2.727 MiB audiofile = clear_memory(audiofile)
33 417.426 MiB 0.000 MiB gc.collect()
34 417.426 MiB 0.000 MiB x = add_to_list(audiofile2, x)
35 425.047 MiB 7.621 MiB audiofile2 = clear_memory(audiofile2)
36 285.344 MiB -139.703 MiB gc.collect()
37 285.344 MiB 0.000 MiB x = add_to_list(audiofile3, x)
38 340.082 MiB 54.738 MiB audiofile3 = clear_memory(audiofile3)
39 339.582 MiB -0.500 MiB gc.collect()
Is it normal for memory_profiler to show half of the memory that's actually being used by the CPU(s)? And is that in fact what is occurring here?
Also note that when gc.collect
is not explicitly called, the -139.703 MiB
freed at line 36
become:
================================================
35 374.895 MiB -45.617 MiB audiofile2 = clear_memory(audiofile2)
In the line-by-line report,
memory_profiler
measures the memory usage after the execution of each line. In the memory peaks inside a function, e.g insidemake_objects
but the memory is released before the function returns, then memory_profiler will not report that usage.Workarounds include decorating also the nested functions (e.g. make_objects) or using mprof (distributed with memory_profiler) to report memory usage as a function of time.