Running tasks parallel in powershell

Question

I have a PowerShell script like this:

Foreach ($file in $files) {
    [Do something]
    [Do something]
    [Do something]
}

This way one file is treated after the other. I want to treat 4 files at the same time.

I know of the foreach -parallel loop, but that does the [do something] tasks in parallel. I basically want to run the whole foreach loop in parallel.

How can I achieve this in PowerShell?

Answer 1

You might look into Jobs or runspaces. Here is an example of Jobs:

$block = {
    Param([string] $file)
    "[Do something]"
}
#Remove all jobs
Get-Job | Remove-Job
$MaxThreads = 4
#Start the jobs. Max 4 jobs running simultaneously.
foreach($file in $files){
    While ($(Get-Job -state running).count -ge $MaxThreads){
        Start-Sleep -Milliseconds 3
    }
    Start-Job -Scriptblock $Block -ArgumentList $file
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0){
    start-sleep 1
}
#Get information from each job.
foreach($job in Get-Job){
    $info= Receive-Job -Id ($job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job

In the above code I have it where each $file is running in parallel with eachother (Up to 4 running simultaneously).

EDIT: In response to the comments, here is some documentation about scriptblocks. The short reason about why you must include the parameter is because unlike PowerShell functions, scriptblocks can't specify parameters outside of the braces {}.

Answer 2

According to Get-Help about_Foreach-Parallel, ForEach -Parallel... will process the entire scriptblock in parallel for each item, but the commands in the scriptblock will be processed sequentially (though presumably they will be parallelized if bracketed with Parallel {...}). However, your script must be a PowerShell workflow for this to be accepted; the Parallel and Sequence keywords are only effective within workflows.