Multiprocessing: Pool and pickle Error — Pickling

Question

I have two files:

x.py

class BF(object)
   def __init__():
   .
   .
   def add(self,z):
   .
   .

y.py

from y import BF
def FUNC((a,b,bf))
   .
   .
   bf.add(x)
   .
   .
   return bf

.
.
if __name__ == '__main__':
    pool = multiprocessing.Pool(3)
    for i in range(len(sl)):
         bf_set.append(BF())
    results = pool.map(FUNC,zip(sl, itertools.repeat(aa), bf_set))

I also tried to define BF inside FUNC, but sill I got:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

I've read some posts for related issues, but they have their pool.map() inside of the class, so the solutions cannot be applied to this problem (i guess).

Any idea?

Answer 1

I'm going to basically use what you have above, but turn it into working code. There is no problem serializing, if you use dill. I'm using a fork of multiprocessing called pathos.multiprocessing, which uses dill instead of pickle.

>>> def FUNC((a,b,bf)):
...   z = a+b
...   bf.add(z)
...   return bf
... 
>>> class BF(object):
...   def add(self, z):
...     self.z += z
...   def __init__(self):
...     self.z = 0
... 
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> pool = Pool()
>>> 
>>> f = BF()
>>> f.add(1)
>>> f.z
1
>>> 
>>> FUNC((0,1,f))
<__main__.BF object at 0x10d387f50>
>>> 
>>> FUNC((0,1,f)).z
2
>>> 
>>> sl = [BF() for i in range(10)]
>>> results = pool.map(FUNC, zip(range(len(sl)), range(len(sl)), sl))
>>> [bf.z for bf in results]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

This works, because pathos uses dill, which can serialize almost anything in python.

>>> import dill as pickle
>>> pickle.loads(pickle.dumps(bf.add))
<bound method BF.add of <__main__.BF object at 0x10d383950>>
>>> pickle.loads(pickle.dumps(BF.add))
<unbound method BF.add>

Get pathos and dill at: https://github.com/uqfoundation