Pickle can't store an object in django locmem

Question

Something that's puzzling me a bit...

>>> from django.core.cache import get_cache
>>>
>>> cache = get_cache('django.core.cache.backends.locmem.LocMemCache')
>>>
>>> # Set the 'content' cache key to a string
>>> cache.set('content', 'a string')
>>> cache.get('content')
'a string'
>>>
>>> class TestObj(object):
...     pass
>>>
>>> a = TestObj()
>>> cache.set('content', a)
>>>
>>> # cache hasn't updated...
>>> cache.get('content')
'a string'
>>>
>>> cache.set('content', 1)
>>> # this is fine however..
>>> cache.get('content')
1
>>>

Ok, so the cache doesn't accept objects for some reason.

# in locmem.py, set() method
try:
    pickled = pickle.dumps(new_value, pickle.HIGHEST_PROTOCOL)
    self._cache[key] = pickled
except pickle.PickleError:
    pass

This will be why, it's obviously hitting the PickleError

>>> import pickle
>>> pickled = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
    save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'TestObj'>: it's not found as __builtin__.TestObj

Sure thing, but why is this happening? It works just fine in the python console, but not the django shell?

# Works fine in python shell...
>>> import pickle  
>>> class TestObj(object):
...     pass
... 
>>> testobj = TestObj()   
>>> pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)
>>> pickled
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'
>>>

This problem arose because I'm trying to store a Mock() object in the cache for a test. Not sure if I'm going about this the wrong way...

Answer 1

WIth help from Martijn in this follow-up question it turns out the short answer is:

"Yup".

You can't pickle Mock() objects as they do not provide the top-level object that they are mocking so pickle therefore has no idea where to import from. As the cache requires the object to be pickled to store it, it is not possible to store a Mock() instance in the LocMemCache. Will have to rethink how I go about testing this.

Answer 2

The problem is that pickle serializes classes by reference, so could you not just use a better serializer that pickles the class by serializing the class definitions instead of by reference? Then you'd pickle a mock object, which would then pickle the class source code, and then you'd be able to pass that to the django cache. I'm the author of dill, which is a better serializer… and also the author of klepto, which is a caching package… and this is exactly what I do to store any object in a SQL table, on disk, or in an in-memory cache.

Essentially (not trying this, but guessing it works based on experience with my own caching package), it should work like this:

>>> from django.core.cache import get_cache
>>> import dill
>>>
>>> cache = get_cache('django.core.cache.backends.locmem.LocMemCache')
>>>
>>> # Set the 'content' cache key to a string
>>> cache.set('content', dill.dumps('a string'))
>>> dill.loads(cache.get('content'))
'a string'
>>>
>>> class TestObj(object):
...     pass
>>>
>>> a = TestObj()
>>> cache.set('content', dill.dumps(a))
>>>
>>> dill.loads(cache.get('content'))
<__main__.TestObj object at 0x10235e510>
>>>
>>> # this is pickling classes w/o using a reference
>>> dill.dumps(a)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x07TestObjq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13b.'
>>> # and here's using a reference, which is exactly how pickle does it
>>> dill.dumps(a, byref=True)
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'

If you want to try it yourself, get dill (and klepto) here: https://github.com/uqfoundation

Answer 3

It happens because django LocMemCache uses cPickle instead of pickle by default. You can see it in LocMemCache class:

try:
    from django.utils.six.moves import cPickle as pickle
except ImportError:
    import pickle

If you will try to do in shell:

from django.utils.six.moves import cPickle as pickle
testobj = TestObj()
pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)

It will be same error.

As possible solution I propose you to pack objects manually in your tests using pickle and after that do cache.set():

a = TestObj()
pickled = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
cache.set('content', a)

Answer 4

sample code

import pickle  # or from django.utils.six.moves import cPickle as pickle
lass TestObj(object):
     pass
testobj = TestObj()   
pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)
pickled
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'

i can't understand a behavior, when i open a console session with "python manage.py shell" and execute: _pickle.PicklingError: Can't pickle : attribute lookup builtins.TestObj failed

but when open a single python console and execute the same code, works fine! Is there a reason?

I observed too, if i open a single console, and import from django.utils.six.moves import cPickle as pickle works fine too. The problem just occours when the code is executed in django context. :(