python parallel map (multiprocessing.Pool.map) wit

问题:

I'm trying to call a function on multiple processes. The obvious solution is python's multiprocessing module. The problem is that the function has side effects. It creates a temporary file and registers that file to be deleted on exit using the atexit.register and a global list. The following should demonstrate the problem (in a different context).

import multiprocessing as multi

glob_data=[]
def func(a):
    glob_data.append(a)

map(func,range(10))
print glob_data  #[0,1,2,3,4 ... , 9]  Good.

p=multi.Pool(processes=8)
p.map(func,range(80))

print glob_data  #[0,1,2,3,4, ... , 9] Bad, glob_data wasn't updated.

Is there any way to have the global data updated?

Note that if you try out the above script, you probably shouldn't try it from the interactive interpreter since multiprocessing requires the module __main__ to be importable by child processes.

UPDATE

Added the global keyword in func doesn't help -- e.g.:

def func(a):  #Still doesn't work.
    global glob_data
    glob_data.append(a)

回答1:

You need the list glob_data to be backed by shared memory, Multiprocessing's Manager gives you just that:

import multiprocessing as multi
from multiprocessing import Manager

manager = Manager()

glob_data = manager.list([])

def func(a):
    glob_data.append(a)

map(func,range(10))
print glob_data  # [0,1,2,3,4 ... , 9] Good.

p = multi.Pool(processes=8)
p.map(func,range(80))

print glob_data # Super Good.

For some background:

https://docs.python.org/3/library/multiprocessing.html#managers

回答2:

Have func return a tuple with the results you want from the processing and the thing you want to append to glob_data. Then, when the p.map has completed, you can extract the results from the first elements in the returned tuples and you can build glob_data from the second elements.