PicklingError: Can't pickle : it's

Question

I'm getting a PicklingError from this line of code in my GAE Python app:

deferred.defer(email_voters_begin, ekey, voter_list)

The three arguments are:

• email_voters_begin -- A Python function, eg., function email_voters_begin at 0x1035d4488
• ekey -- A key to an entity I defined, eg., prints as agdvcGF2b3Rlcg4LEghFbGVjdGlvbhgCDA
• voter_list -- A list of objects I defined, eg., [models.Voter object at 0x103d3d310, ... ]

When this line executes as part of my tests (with webtest and nosegae), I get the following error:

Traceback (most recent call last):
  [...]
  File "/Users/joneill/OpenSTV/OpenSTV/trunk/OpaVote-HR/tasks.py", line 29, in init_voters_and_send_email
    deferred.defer(email_voters_begin, ekey, voter_list)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/deferred/deferred.py", line 249, in defer
    pickled = serialize(obj, *args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/deferred/deferred.py", line 221, in serialize
    return pickle.dumps(curried, protocol=pickle.HIGHEST_PROTOCOL)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  [...]
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 396, in save_reduce
    save(cls)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 753, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'google.appengine.ext.blobstore.blobstore.BlobInfo'>: it's not the same object as google.appengine.ext.blobstore.blobstore.BlobInfo

Note that the Voter entities passed in deferred.defer() do not have a BlobReference property, but the Voter entities do have a ReferenceProperty to another entity that does have a BlobReference property. I wouldn't think that any BlobInfo objects would be part of the pickle but the error suggests one is being included.

This error does not occur when I run the same code from a browser window using the dev server.

I'm stumped as how to debug this and any ideas would be greatly appreciated.

Answer 1

This is likely occurring because you've retrieved the ReferenceProperty on at least one of the Voter objects. Once you dereference a ReferenceProperty, the model instance caches it. Pickling pickles any cached objects too, so it attempts to serialize the Voter instance, the referenced instance, and its BlobInfo.

In general, as Skirmantas points out, it's usually a bad idea to pass model instances to deferred. Where possible, send keys, and if not, serialize the instances to Protocol Buffers and send those instead.

Answer 2

You should never pass instances of models to a deferred. Use keys instead:

deferred.defer(email_voters_begin, ekey, [v.key() for v in voter_list])

In your email_voters_begin:

def email_voters_begin(ekey, voters_keys):
    voter_list = models.Voter.get(voters_keys)