I am having a bit of trouble with Parallel processing in Python. I am completely new to the concept of Parallel computing. I use multiprocessing that comes with standard Python.
I have 12 threads in my computer. I ask for 12 workers, but I am not always able to get all workers I ask for. My problem arises when I don't get access to as many workers I need to process nTasks number of tasks in my code below (currently set to four). What happens then is just that the code gets stuck and never gets to what is under the comment "# Get Results". It seems random how many workers I get (I always ask for 12), but the problem arises when I get three workers or less in the following code:
import multiprocessing as mp
import scipy as sp
import scipy.stats as spstat
import pylab
def testfunc(x0, N):
print 'working with x0 = %s' % x0
x = [x0]
for i in xrange(1,N):
x.append(spstat.norm.rvs(size = 1)) # stupid appending to make it slower
if i % 10000 == 0:
print 'x0 = %s, i = %s' % (x0, i)
return sp.array(x)
def testfuncParallel(fargs):
return testfunc(*fargs)
pool = mp.Pool(12) # I have 12 threads
nTasks = 4
N = 100000
tasks = [(x, n) for x, n in enumerate(nTasks*[N])] # nTasks different tasks
result = pool.map(testfuncParallel, tasks)
pool.close()
pool.join()
# Get results:
sim = sp.zeros((N, nTasks))
for nn, res in enumerate(result):
sim[:, nn] = res
pylab.figure()
for i in xrange(nTasks):
pylab.subplot(nTasks,1, i + 1)
pylab.plot(sim[:, i])
pylab.show()
I have tried to use pool.map_async instead of pool.map but I cannot get around the problem.
Thanks in advance,
Sincerely,
Matias