QRunnable in multiple cores

问题:

I am learning about QRunnable and I have the following code:

from PyQt5.QtCore import QThreadPool, QRunnable

class SomeObjectToDoComplicatedStuff(QRunnable):
    def __init__(self, name):
        QRunnable.__init__(self)
        self.name = name

    def run(self):
        print('running', self.name)
        a = 10
        b = 30
        c = 0
        for i in range(5000000):
            c += a**b
        print('done', self.name)


pool = QThreadPool.globalInstance()
pool.setMaxThreadCount(10)

batch_size = 100

workers = [None] * batch_size

for i in range(batch_size):
    worker = SomeObjectToDoComplicatedStuff('object ' + str(i))
    workers[i] = worker
    pool.start(worker)

print('All cued')
pool.waitForDone()

# processing the results back
for i in range(batch_size):
    print(workers[i].name, ' - examining again.')

I see that indeed there are different processes being alternated, but all is happening on a single core.

How can I make this code run using all the processor cores?

PS: This code is just a simplification of a super complicated number crunching application I am making. In it, I want to to do Monte Carlo in several threads and the worker itself is a complex optimization problem. I have tried the python multiprocessing module but it doesn't handle scipy too well.

回答1:

Not sure how much use this will be, but a multiprocessing version of your example script would be something like this:

from multiprocessing import Pool

class Worker(object):
    def __init__(self, name):
        self.name = name

    def run(self):
        print('running', self.name)
        a = 10
        b = 30
        c = 0
        for i in range(5000000):
            c += a**b
        print('done', self.name)
        return self.name, c

def caller(worker):
    return worker.run()

def run():
    pool = Pool()
    batch_size = 10
    workers = (Worker('object%d' % i) for i in range(batch_size))
    result = pool.map(caller, workers)
    for item in result:
        print('%s = %s' % item)

if __name__ == '__main__':

    run()

回答2:

How can I make this code run using all the processor cores?

Using PyQt (QRunner/QThread and likely), I think it's almost impossible because they (the python version, not the C++) are using the GIL.

The easiest solution would be to use multiprocessing, but since you have some problem using it along scipy you should look for some non-standard library.

I suggest you to take a look at ipyparallel, AFAIK they're developed under the same umbrella, so they're likely to work seamlessy.