Python 2.7 multiprocessing get process result whit

Question

How can I get the result from my process without using a pool ?

(I'm willing to conserve an eye on the progression:

(print "\r",float(done)/total,"%",)

which can't be done using a pool as far I know)

def multiprocess(function, argslist, ncpu):
    total = len(argslist)
    done = 0
    jobs = []
    while argslist != []:
        if len(mp.active_children()) < ncpu:
            p = mp.Process(target=function,args=(argslist.pop(),))
            jobs.append(p)
            p.start()
            done+=1
            print "\r",float(done)/total,"%",
    #get results here
    for job in jobs:
        job.get_my_result()???

The processes are really short (<0.5 seconds) but I have around 1 million of them.

I saw this thread Can I get a return value from multiprocessing.Process? I tried to reproduce it but I couldn't make it work properly.

At your entire disposal for any further information.

Answer 1

This question may be considered as a duplicate but anyway here is the solution to my problem:

def multiprocess(function, argslist, ncpu):
    total = len(argslist)
    done = 0
    result_queue = mp.Queue()
    jobs = []
    while argslist != [] and done<10 :
        if len(mp.active_children()) < ncpu:
            p = mp.Process(target=function,args=(result_queue, argslist.pop(),))
            jobs.append(p)
            p.start()
            done+=1
            print "\r",float(done)/total,"%",
    #get results here
    res = [result_queue.get() for p in jobs]
    print res

and I had to change as well the

return function_result

into

result_queue.put(function_result)

Answer 2

The easiest way should be a queue that is passed as argument to your function. The results of that function can be put into that queue and later on you can iterate over that queue to collect all the results or process it as soon as a result arrives. However, it only works when the you can work with "unordered" results. See the Python documentation for details: Examples for Multiprocessing and Queues