How to apply a function in parallel to multiple im

Question

Let's say I have an images array that holds 100,000 images with 3 channels.

images = np.random.randint(0,255,(100000,32,32,3))

And I have a function foo which accepts an image and performs some operation on it.

def foo(img):
    #some operation on the image, say histogram equalization

How do I now apply the foo function to 100000 images in parallel? I thought numpy would have some function for this purpose, but I was disappointed to not find any. I found numpy.apply_along_axis but I read it is rather iterative. What should I do?

Answer 1

Here is an example, using joblib which performs histogram equalization on the images, in parallel with n_jobs equal to nprocs (here 10 processes but you can change as per your need)

# imports
import numpy as np
from skimage import exposure
from joblib import Parallel, delayed

# number of processes
nprocs = 10

# batched image array
img_arr = np.random.randint(0, 255, (1000, 32, 32, 3))

# function to be applied on all images
def process_image(img):
     img_eq = exposure.equalize_hist(img)
     return img_eq

result = []

# run `process_image()` in parallel
result.extend(Parallel(n_jobs=nprocs)(delayed(process_image)(img_arr[idx]) for idx in range(img_arr.shape[0])))