Python Multiprocessing Lib Error (AttributeError:

Question

Am getting this error when using the pool.map(funct, iterable):

AttributeError: __exit__

No Explanation, only stack trace to the pool.py file within the module.

using in this way:

with Pool(processes=2) as pool:
   pool.map(myFunction, mylist)
   pool.map(myfunction2, mylist2)

I suspect there could be a problem with the picklability (python needs to pickle, or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.

EDIT: new format of code that produces this error :

def governingFunct(list):
    #some tasks
    def myFunction():
         # function contents
    with closing(Pool(processes=2)) as pool:
         pool.map(myFunction, sublist)
         pool.map(myFunction2, sublist2)

ERROR PRODUCED:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Answer 1

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

For earlier Python versions, you could use contextlib.closing(), but take into account this'll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)

Answer 2

with statement is for object that have __enter__ and __exit__ functions, i.e. Context Manager Types
multiprocessing.Pool is not Context Manager Type. try do the following:

pool = Pool(processes=2)
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)