Pool within a Class in Python

Question

I would like to use Pool within a class, but there seems to be a problem. My code is long, I created a small-demo variant to illustrated the problem. It would be great if you can give me a variant of the code below that works.

from multiprocessing import Pool

class SeriesInstance(object):
    def __init__(self):
        self.numbers = [1,2,3]
    def F(self, x):
        return x * x
    def run(self):
        p = Pool()
        print p.map(self.F, self.numbers)


ins = SeriesInstance()
ins.run()

Outputs:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

And then hangs.

Answer 1

It looks like because of the way the function gets passed to the worker threads (pickling) you can't use instance methods unfortunately. My first thought was to use lambdas, but it turns out the built in pickler can't serialize those either. The solution, sadly, is just to use a function in the global namespace. You can still make it an instance attribute though, take a look:

from multiprocessing import Pool

def F(x):
    return x * x

class SeriesInstance(object):
    def __init__(self):
        self.numbers = [1,2,3]
        self.F = F

    def run(self):
        p = Pool()
        out = p.map(self.F, self.numbers)
        p.close()
        p.join()
        return out

if __name__ == '__main__':
    print SeriesInstance().run()