memory leaks using matplotlib

Question

This is not intended as a bug report--even if these leaks may be a result of mpl bugs, please interpret the question ask asking for a way around them.

The problem is simple: plot a large chunk of data (using plot() or scatter()), clear/release everything, garbage collect, but still not nearly all the memory is released.

Line #    Mem usage    Increment   Line Contents
================================================
391  122.312 MiB    0.000 MiB   @profile
392                             def plot_network_scatterplot(t_sim_stop, spikes_mat, n_cells_per_area, n_cells, basedir_output, condition_idx):
393
394                                  # make network scatterplot
395  122.312 MiB    0.000 MiB        w, h = plt.figaspect(.1/(t_sim_stop/1E3))
396  122.324 MiB    0.012 MiB        fig = mpl.figure.Figure(figsize=(10*w, 10*h))
397  122.328 MiB    0.004 MiB        canvas = FigureCanvas(fig)
398  122.879 MiB    0.551 MiB        ax = fig.add_axes([.01, .1, .98, .8])
399  134.879 MiB   12.000 MiB        edgecolor_vec = np.array([(1., 0., 0.), (0., 0., 1.)])[1-((spikes_mat[:,3]+1)/2).astype(np.int)]
400                                  '''pathcoll = ax.scatter(spikes_mat[:,1],
401                                             spikes_mat[:,0] + n_cells_per_area * (spikes_mat[:,2]-1),
402                                             s=.5,
403                                             c=spikes_mat[:,3],
404                                             edgecolor=edgecolor_vec)'''
405  440.098 MiB  305.219 MiB        pathcoll = ax.plot(np.random.rand(10000000), np.random.rand(10000000))
406  440.098 MiB    0.000 MiB        ax.set_xlim([0., t_sim_stop])
407  440.098 MiB    0.000 MiB        ax.set_ylim([1, n_cells])
408  440.098 MiB    0.000 MiB        plt.xlabel('Time [ms]')
409  440.098 MiB    0.000 MiB        plt.ylabel('Cell ID')
410  440.098 MiB    0.000 MiB        plt.suptitle('Network activity scatterplot')
411                                  #plt.savefig(os.path.join(basedir_output, 'network_scatterplot-[cond=' + str(condition_idx) + '].png'))
412  931.898 MiB  491.801 MiB        canvas.print_figure(os.path.join(basedir_output, 'network_scatterplot-[cond=' + str(condition_idx) + '].png'))
413                                  #fig.canvas.close()
414                                  #pathcoll.set_offsets([])
415                                  #pathcoll.remove()
416  931.898 MiB    0.000 MiB        ax.cla()
417  931.898 MiB    0.000 MiB        ax.clear()
418  931.898 MiB    0.000 MiB        fig.clf()
419  931.898 MiB    0.000 MiB        fig.clear()
420  931.898 MiB    0.000 MiB        plt.clf()
421  932.352 MiB    0.453 MiB        plt.cla()
422  932.352 MiB    0.000 MiB        plt.close(fig)
423  932.352 MiB    0.000 MiB        plt.close()
424  932.352 MiB    0.000 MiB        del fig
425  932.352 MiB    0.000 MiB        del ax
426  932.352 MiB    0.000 MiB        del pathcoll
427  932.352 MiB    0.000 MiB        del edgecolor_vec
428  932.352 MiB    0.000 MiB        del canvas
429  505.094 MiB -427.258 MiB        gc.collect()
430  505.094 MiB    0.000 MiB        plt.close('all')
431  505.094 MiB    0.000 MiB        gc.collect()

I have tried many combinations and different orders of all the clear/release to no avail. I've tried not using an explicit fig/canvas creation but just using mpl.pyplot, with the same results.

Is there any way to free this memory, and go out with the 122.312 that I came in?

Cheers!

Answer 1

Alex Martelli explains

It's very hard, in general, for a process to "give memory back to the OS" (until the process terminates and the OS gets back all the memory, of course) because (in most implementation) what malloc returns is carved out of big blocks for efficiency, but the whole block can't be given back if any part of it is still in use." So what you think is a memory leak may just be a side effect of this. If so, fork can solve the problem.

Furthermore,

The only really reliable way to ensure that a large but temporary use of memory DOES return all resources to the system when it's done, is to have that use happen in a subprocess, which does the memory-hungry work then terminates."

Therefore, you instead of trying to clear the figure and axes, delete references and garbage collecting (all of which will not work), you can instead use multiprocessing to run plot_network_scatterplot in a separate process:

import multiprocessing as mp

def plot_network_scatterplot(
    t_sim_stop, spikes_mat, n_cells_per_area, n_cells, basedir_output, 
    condition_idx):

    # make network scatterplot
    w, h = plt.figaspect(.1/(t_sim_stop/1E3))
    fig = mpl.figure.Figure(figsize=(10*w, 10*h))
    canvas = FigureCanvas(fig)
    ax = fig.add_axes([.01, .1, .98, .8])
    edgecolor_vec = np.array([(1., 0., 0.), (0., 0., 1.)])[1-((spikes_mat[:,3]+1)/2).astype(np.int)]
    '''pathcoll = ax.scatter(spikes_mat[:,1],
               spikes_mat[:,0] + n_cells_per_area * (spikes_mat[:,2]-1),
               s=.5,
               c=spikes_mat[:,3],
               edgecolor=edgecolor_vec)'''
    pathcoll = ax.plot(np.random.rand(10000000), np.random.rand(10000000))
    ax.set_xlim([0., t_sim_stop])
    ax.set_ylim([1, n_cells])
    plt.xlabel('Time [ms]')
    plt.ylabel('Cell ID')
    plt.suptitle('Network activity scatterplot')
    canvas.print_figure(os.path.join(basedir_output, 'network_scatterplot-[cond=' + str(condition_idx) + '].png'))

def spawn(func, *args):
    proc = mp.Process(target=func, args=args)
    proc.start()
    # wait until proc terminates.
    proc.join()

if __name__ == '__main__':
    spawn(plot_network_scatterplot, t_sim_stop, spikes_mat, n_cells_per_area, 
          n_cells, basedir_output, condition_idx)