PicklingError: Can't pickle : it's

Question

This is the error I got today at http://filmaster.com">filmaster.com:

PicklingError: Can't pickle : it's not the same object as decimal.Decimal

What does that exactly mean? It does not seem to be making a lot of sense... It seems to be connected with django caching. You can see the whole traceback here:

Traceback (most recent call last):

File "/home/filmaster/django-trunk/django/core/handlers/base.py", line 92, in get_response response = callback(request, *callback_args, **callback_kwargs)

File "/home/filmaster/film20/film20/core/film_views.py", line 193, in show_film
workflow.set_data_for_authenticated_user()

File "/home/filmaster/film20/film20/core/film_views.py", line 518, in set_data_for_authenticated_user
object_id = self.the_film.parent.id)

File "/home/filmaster/film20/film20/core/film_helper.py", line 179, in get_others_ratings
set_cache(CACHE_OTHERS_RATINGS, str(object_id) + "_" + str(user_id), userratings)

File "/home/filmaster/film20/film20/utils/cache_helper.py", line 80, in set_cache return cache.set(CACHE_MIDDLEWARE_KEY_PREFIX + full_path, result, get_time(cache_string))

File "/home/filmaster/django-trunk/django/core/cache/backends/memcached.py", line 37, in set
self._cache.set(smart_str(key), value, timeout or self.default_timeout)

File "/usr/lib/python2.5/site-packages/cmemcache.py", line 128, in set val, flags = self._convert(val)

File "/usr/lib/python2.5/site-packages/cmemcache.py", line 112, in _convert val = pickle.dumps(val, 2)

PicklingError: Can't pickle : it's not the same object as decimal.Decimal

And the source code for Filmaster can be downloaded from here: bitbucket.org/filmaster/filmaster-test

Any help will be greatly appreciated.

Answer 1

I will demonstrate the problem with simple Python classes in Python2.7:

In [13]: class A: pass  
In [14]: class B: pass

In [15]: A
Out[15]: <class __main__.A at 0x7f4089235738>

In [16]: B
Out[16]: <class __main__.B at 0x7f408939eb48>

In [17]: A.__name__ = "B"

In [18]: pickle.dumps(A)
---------------------------------------------------------------------------
PicklingError: Can't pickle <class __main__.B at 0x7f4089235738>: it's not the same object as __main__.B

This error is shown because we are trying to dump A, but because we changed its name to refer to another object "B", pickle is actually confused with which object to dump - class A or B. Apparently, pickle guys are very smart and they have already put a check on this behavior.

Solution: Check if the object you are trying to dump has conflicting name with another object.

I have demonstrated debugging for the case presented above with ipython and ipdb below:

PicklingError: Can't pickle <class __main__.B at 0x7f4089235738>: it's not the same object as __main__.B

In [19]: debug
> /<path to pickle dir>/pickle.py(789)save_global()
    787                 raise PicklingError(
    788                     "Can't pickle %r: it's not the same object as %s.%s" %
--> 789                     (obj, module, name))
    790
    791         if self.proto >= 2:

ipdb> pp (obj, module, name)               **<------------- you are trying to dump obj which is class A from the pickle.dumps(A) call.**
(<class __main__.B at 0x7f4089235738>, '__main__', 'B')
ipdb> getattr(sys.modules[module], name)   **<------------- this is the conflicting definition in the module (__main__ here) with same name ('B' here).**
<class __main__.B at 0x7f408939eb48>

I hope this saves some headaches! Adios!!

Answer 2

I can't explain why this is failing either, but my own solution to fix this was to change all my code from doing

from point import Point

to

import point

this one change and it worked. I'd love to know why... hth

Answer 3

I got this error when running in an jupyter notebook. I think the problem was that I was using %load_ext autoreload autoreload 2. Restarting my kernel and rerunning solved the problem.

Answer 4

One oddity of Pickle is that the way you import a class before you pickle one of it's instances can subtly change the pickled object. Pickle requires you to have imported the object identically both before you pickle it and before you unpickle it.

So for example:

from a.b import c
C = c()
pickler.dump(C)

will make a subtly different object (sometimes) to:

from a import b
C = b.c()
pickler.dump(C)

Try fiddling with your imports, it might correct the problem.

Answer 5

Did you somehow reload(decimal), or monkeypatch the decimal module to change the Decimal class? These are the two things most likely to produce such a problem.

Answer 6

There can be issues starting a process with multiprocessing by calling __init__. Here's a demo:

import multiprocessing as mp

class SubProcClass:
    def __init__(self, pipe, startloop=False):
        self.pipe = pipe
        if startloop:
            self.do_loop()

    def do_loop(self):
        while True:
            req = self.pipe.recv()
            self.pipe.send(req * req)

class ProcessInitTest:
    def __init__(self, spawn=False):
        if spawn:
            mp.set_start_method('spawn')
        (self.msg_pipe_child, self.msg_pipe_parent) = mp.Pipe(duplex=True)

    def start_process(self):
        subproc = SubProcClass(self.msg_pipe_child)
        self.trig_proc = mp.Process(target=subproc.do_loop, args=())
        self.trig_proc.daemon = True
        self.trig_proc.start()

    def start_process_fail(self):
        self.trig_proc = mp.Process(target=SubProcClass.__init__, args=(self.msg_pipe_child,))
        self.trig_proc.daemon = True
        self.trig_proc.start()

    def do_square(self, num):
        # Note: this is an synchronous usage of mp,
        # which doesn't make sense. But this is just for demo
        self.msg_pipe_parent.send(num)
        msg = self.msg_pipe_parent.recv()
        print('{}^2 = {}'.format(num, msg))

Now, with the above code, if we run this:

if __name__ == '__main__':
    t = ProcessInitTest(spawn=True)
    t.start_process_fail()
    for i in range(1000):
        t.do_square(i)

We get this error:

Traceback (most recent call last):
  File "start_class_process1.py", line 40, in <module>
    t.start_process_fail()
  File "start_class_process1.py", line 29, in start_process_fail
    self.trig_proc.start()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 274, in _Popen
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/popen_spawn_posix.py", line 33, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/popen_spawn_posix.py", line 48, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function SubProcClass.__init__ at 0x10073e510>: it's not the same object as __main__.__init__

And if we change it to use fork instead of spawn:

if __name__ == '__main__':
    t = ProcessInitTest(spawn=False)
    t.start_process_fail()
    for i in range(1000):
        t.do_square(i)

We get this error:

Process Process-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
TypeError: __init__() missing 1 required positional argument: 'pipe'

But if we call the start_process method, which doesn't call __init__ in the mp.Process target, like this:

if __name__ == '__main__':
    t = ProcessInitTest(spawn=False)
    t.start_process()
    for i in range(1000):
        t.do_square(i)

It works as expected (whether we use spawn or fork).