Python: Can't pickle type X, attribute lookup

问题:

I am trying to pickle a namedtuple:

from collections import namedtuple
import cPickle

class Foo:

    Bar = namedtuple('Bar', ['x', 'y'])

    def baz(self):
        s = set()
        s.add(Foo.Bar(x=2, y=3))
        print cPickle.dumps(s)

if __name__ == '__main__':
    f = Foo()
    f.baz()

This produces the following output:

Traceback (most recent call last):
  File "scratch.py", line 15, in <module>
    f.baz()
  File "scratch.py", line 11, in baz
    print cPickle.dumps(s)
cPickle.PicklingError: Can't pickle <class '__main__.Bar'>: attribute lookup __main__.Bar failed

What am I doing wrong? Is the problem that Bar is a member of Foo? (Moving the definition of Bar to the top level solves the problem, although I'm still curious why this happens.)

回答1:

Yes, the fact that it's a class member is a problem:

>>> class Foo():
...     Bar = namedtuple('Bar', ['x','y'])
...     def baz(self):
...         b = Foo.Bar(x=2, y=3)
...         print(type(b))
...
>>> a = Foo()
>>> a.baz()
<class '__main__.Bar'>

The problem is that when namedtuple() returns a type object, it isn't aware of the fact that it's being assigned to a class member - and thus, it tells the type object that its type name should be __main__.Bar, even though it should really be __main__.Foo.Bar.

回答2:

Nesting classes makes pickle fail, since it relies on the path of the object inside your application to reconstruct it later.

The immediate solution is to not nest classes, i.e. move Bar definition to outside Foo. Code will work all the same.

But a better thing to do is to not use pickle at all to store data. Use some other serialization format, like json, or a database, like sqlite3.

You have just hit one of the many inconveniences of pickle, if you change your code, move things around, or sometimes make small structural changes, your data becomes unloadable.

Besides that, pickle has other disadvantages: It is slow, unsecure, python-only...

回答3:

Using dill in place of pickle here will allow this to work

回答4:

The solution here is to move your named tuple definition to the module level, then pickle works. A detailed answer is provided here:

How to pickle a namedtuple instance correctly