Mysterious pickle error while profiling a multi-pr

Question

I"m using the multiprocessing module, and I'm using UpdateMessage objects (my own class), sent through multiprocessing.Queue objects, to communicate between processes. Here's the class:

class UpdateMessage:
    def __init__(self, arrayref, rowslice, colslice, newval):
        self.arrayref = arrayref
        self.rowslice = rowslice
        self.colslice = colslice
        self.newval = newval
    def do_update(self):
        if self.arrayref == 'uL':
            arr = uL
        elif self.arrayref == 'uR':
            arr = uR
        else:
            raise Exception('UpdateMessage.arrayref neither uL nor uR')
        arr[self.rowslice, self.colslice] = self.newval

When I run the script, it works perfectly fine. However, when I run it with either cProfile or profile, it gives the following error:

_pickle.PicklingError: Can't pickle <class '__main__.UpdateMessage'>: attribute lookup __main__.UpdateMessage failed

It seems to be trying to pickle the class, but I can't see why this is happening. My code doesn't do this, and it works just fine without it, so it's probably the multiprocessing module. But why would it need to pickle UpdateMessage, and how can I fix the error?

EDIT: here's part of the code that's sending the UpdateMessage (multiple parts of the script do this, but all in the same way):

msg = UpdateMessage(uLref, refer[0] + marker[0] - 2,
                    slice(uL.shape[1]), ustar.copy())
queue.put(msg)

The traceback isn't very helpful:

Traceback (most recent call last):
  File "/usr/lib/python3.2/multiprocessing/queues.py", line 272, in _feed
    send(obj)

Answer 1

I'm assuming the pickling attempt on your class is happening before your class is sent to another process. The easiest solution is probably just to implement the pickle-protocol explicitly on your class...

Answer 2

I had the same problem, and solved it by defining the class to be pickled in its own file.

Answer 3

I do not know how your processes look like but:

'__main__.UpdateMessage'

refers to UpdateMessage in the launched module.

When another process is started not with the Module of the class UpdateMessage, UpdateMessage will not be available.

you have to import that module that includes UpdateMessage so UpdateMessage.__module__ is not '__main__'.

Then Pickle can find UpdateMessage also in other programs.

I recommend the __main__.py to look like this:

import UpdateMessage_module

UpdateMessage_module.main()