ThreadPool and Pool for parallel processing

Question

Is there a way to use both ThreadPool and Pool in python to parallelise a loop by specifying the number of CPUs and cores you wish to use?

For example I would have a loop execute as:

from multiprocessing.dummy import Pool as ThreadPool 
from tqdm import tqdm
import numpy as np

def my_function(x):
    return x + 1

pool = ThreadPool(4)
my_array = np.arange(0,1e6,1)


results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array)))

For 4 cores but it I wanted to spread these out on multiple CPUs as well, is there a simple way to adapt the code?

Answer 1

You are confusing between a core and a CPU. Generally, for all purposes both are considered to be the same(let's call them processor from now on).

When creating a thread pool in python, the threads are user level threads and are run on the same processor, due to Global Interpreter Lock(GIL) in python. As only one thread can control the python interpreter at a time. So, using (python)threads we don't get any real concurrency in data-intensive tasks.

How to solve this? Easy. Spawn multiple python processes running on different processors(each with its own interpreter). This is where the multi processing(mp) module is used, to spawn multiple processes from the parent python process in which it is called.

You can verify this by running htop(on linux, mac) and analysing the number of python processes. In case of mp module, they all will have the same name as the parent script where the pool.map function is called.

• Timings for your code on a 8 core mac: 39.7s
• Timing for this code on the same machine : 2.9s(note I can use 8 cores at max, but for comparison purposes using only 4)

Below is the modified code:

from multiprocessing.dummy import Pool as ThreadPool 
from tqdm import tqdm
import numpy as np
import time
import multiprocessing as mp

def my_function(x):
    return x + 1

pool = ThreadPool(4)
my_array = np.arange(0,1e6,1)


t1 = time.time()
# results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array)))
pool = mp.Pool(processes=4) # Generally, set to 2*num_cores you have
res = pool.map(my_function, my_array)
print("Time taken = ", time.time() - t1)

Answer 2

multiprocessing.dummy.Pool is exactly simple ThreadPool, which don't use multicores and multicpus (because of GIL). you must use multiprocessing.Pool to run Process, which is process in your OS (if you define Pool(N) - N is number of this processes, if no - number of your cores in OS is default). Arguments this processes get from internal queue of Pool. 'case of that U will use all cpu and all core in your OS