I'm trying to implement metaclass/decorator facilities to allow easy-ish parallelization of code. I'm using Python's multiprocessing.
Say I have:
class Worker(metaclass=Parallelizable):
def __init__(self):
super().__init__()
# annotate some method for parallele computation
@ParalleleMethod
def long_calculation(self, data):
# do stuff
return ans
class ParalleleMethod:
def __init__(self, func):
self.func = func
def __call__(self, data):
# as a prototype of the idea I want to get to
pool.starmap(self.func, data)
When pool.starmap() executes, I get:
_pickle.PicklingError:
Can't pickle <function Worker.long_calculation at 0x7fdbccd4bea0>:
it's not the same object as __main__.Worker.long_calculation
I'm guessing that this is the reason:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. 2 This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.
Since I passed the function to init as an object, that object isn't fully qualified anymore.
Is there a way that I could "add" that full qualification again to self.func? The assumption being that if I were able to do that, pickling would work - it would find the definition it needs to pickle (and unpickle in the other process)?
