Python Memory Leak with Struct and Numpy

Question

Continuation of Python Memory Leak Using binascii, zlib, struct, and numpy but with example code that correctly illustrates the issue I have.

import struct
import zlib
import binascii
import numpy as np
import os
import psutil
import gc

l = list()
l.append('eJztwYEAAAAAw6D5U1/gCFUB' + 'A'*161 + 'McA6vIAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'HwGw6IAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'LwGw6QAAQ==')

process = psutil.Process(os.getpid())
for s in l:
    print (process.get_memory_info()[0] / float(2 ** 20))
    byte_array = zlib.decompress(binascii.a2b_base64(s))
    array = np.array(struct.unpack('%dB' % (len(byte_array)), byte_array))
    del byte_array
    del array
    gc.collect()
    print (process.get_memory_info()[0] / float(2 ** 20))

del l
del s
gc.collect()
print (process.get_memory_info()[0] / float(2 ** 20))

It prints:

22.37109375
25.83203125
25.83203125
95.65625
95.65625
166.69140625
166.69140625

Why does the memory used continue to increase? Why is so much memory used at the end of the script even after the variables are deleted? Thank you.

Answer 1

This link http://bugs.python.org/issue14596 was very helpful. The issue was with the struct module caching format strings. If i explicitly create a Struct object, use it, and then delete it the issue goes away.

import struct
import zlib
import binascii
import os
import psutil
import gc


def print_memory(string):
    print string + str(process.get_memory_info()[0] / float(2 ** 20))

l = list()
l.append('eJztwYEAAAAAw6D5U1/gCFUB' + 'A'*161 + 'McA6vIAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'HwGw6IAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'LwGw6QAAQ==')

process = psutil.Process(os.getpid())
for s in l:
    print_memory('Before inflating: ')
    byte_array = zlib.decompress(binascii.a2b_base64(s))
    _struct = struct.Struct('%dB' % (len(byte_array)))
    array = _struct.unpack(byte_array)
    del byte_array
    del array
    del _struct
    gc.collect()
    print_memory('After inflating and deleting: ')

del l
del s
gc.collect()
print_memory('After deleting everything: ')