Using heapy to track down memory leaks in Django a

Question

I've followed excellent post here how to setup heapy with Django: http://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/

I've commanded hp.setref() and now after awhile I get also data with hp.heap():

>>> hp.heap()
Partition of a set of 12075 objects. Total size = 1515496 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0   4048  34   339656  22    339656  22 str
     1   3112  26   269368  18    609024  40 tuple
     2    171   1   169992  11    779016  51 dict (no owner)
     3   1207  10   144440  10    923456  61 list
     4     49   0   102040   7   1025496  68 dict of module
     5    591   5    66984   4   1092480  72 unicode
     6    498   4    59760   4   1152240  76 function
     7    433   4    51960   3   1204200  79 types.CodeType
     8     57   0    50480   3   1254680  83 type
     9     36   0    31584   2   1286264  85 dict of class

But what now? What I should understand from this output? How I should start tracking down where those 'str' and 'tuple' objects belong?

With get_rp, I get following output:

>>> hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
 0: _ --- [-] 12000 (0xd1d340 | 0xd88b50 | 0xf63f00 | __builtin__.Struct | __...
 1: a      [-] 137 dict (no owner): 0x761c30*160, 0x7655d0*1491, 0x781640*9...
 2: aa ---- [-] 45 dict of django.db.models.options.Options: 0xcf3110...
 3: a3       [-] 45 django.db.models.options.Options: 0xcf3110, 0xf0bb10...
 4: a4 ------ [-] 140 dict of django.db.models.related.RelatedObject: 0x10bec...
 5: a5         [-] 140 django.db.models.related.RelatedObject: 0xf14450...
 6: a6 -------- [-] 63 dict of django.db.models.fields.related.ForeignKey: 0x...
 7: a7           [+] 63 django.db.models.fields.related.ForeignKey: 0xf0e690...
 8: a5b ------- [-] 7 dict of django.db.models.fields.related.OneToOneField: ...
 9: a5ba         [+] 7 django.db.models.fields.related.OneToOneField: 0x15447...

Is the correct assumption now that it's Django that is leaking memory? But what are those dict's who don't have owner?

Answer 1

I don't have any experience with heapy, but in my experience, Django (and most other Python programs) don't leak memory, but they also don't clean up memory as pristinely as some would like.

Also, Django has settings that cause it to consume memory for diagnostic reasons. For example, setting DEBUG=True can cause it to hold on to all SQL queries, so the longer the process runs, the more memory it uses.

UPDATE: Your problem isn't in your Python code. Look at the summary heapy is giving you: the total size of memory represented there is 1.5Mb! When Python programs truly leak, the most common cause is a leaky C extension. Do you have any C extensions that you are running under your Django process?