Can Powershell Run Commands in Parallel?

Question

I have a powershell script to do some batch processing on a bunch of images and I'd like to do some parallel processing. Powershell seems to have some background processing options such as start-job, wait-job, etc, but the only good resource I found for doing parallel work was writing the text of a script out and running those (PowerShell Multithreading)

Ideally, I'd like something akin to parallel foreach in .net 4.

Something pretty seemless like:

foreach-parallel -threads 4 ($file in (Get-ChildItem $dir))
{
   .. Do Work
}

Maybe I'd be better off just dropping down to c#...

Answer 1

http://gallery.technet.microsoft.com/scriptcenter/Invoke-Async-Allows-you-to-83b0c9f0

i created an invoke-async which allows you do run multiple script blocks/cmdlets/functions at the same time. this is great for small jobs (subnet scan or wmi query against 100's of machines) because the overhead for creating a runspace vs the startup time of start-job is pretty drastic. It can be used like so.

with scriptblock,

$sb = [scriptblock] {param($system) gwmi win32_operatingsystem -ComputerName $system | select csname,caption} 

$servers = Get-Content servers.txt 

$rtn = Invoke-Async -Set $server -SetParam system  -ScriptBlock $sb

just cmdlet/function

$servers = Get-Content servers.txt 

$rtn = Invoke-Async -Set $servers -SetParam computername -Params @{count=1} -Cmdlet Test-Connection -ThreadCount 50

Answer 2

Backgrounds jobs are expensive to setup and are not reusable. PowerShell MVP Oisin Grehan has a good example of PowerShell multi-threading http://www.nivot.org/2009/01/22/CTP3TheRunspaceFactoryAndPowerShellAccelerators.aspx

(10/25/2010 site is down):

I'e used adapted Oisin script for use in a data loading routine here:

http://rsdd.codeplex.com/SourceControl/changeset/view/a6cd657ea2be#Invoke-RSDDThreaded.ps1

Answer 3

The answer from Steve Townsend is correct in theory but not in practice as @likwid pointed out. My revised code takes into account the job-context barrier--nothing crosses that barrier by default! The automatic $_ variable can thus be used in the loop but cannot be used directly within the script block because it is inside a separate context created by the job.

To pass variables from the parent context to the child context, use the -ArgumentList parameter on Start-Job to send it and use param inside the script block to receive it.

cls
# Send in two root directory names, one that exists and one that does not.
# Should then get a "True" and a "False" result out the end.
"temp", "foo" | %{

  $ScriptBlock = {
    # accept the loop variable across the job-context barrier
    param($name) 
    # Show the loop variable has made it through!
    Write-Host "[processing '$name' inside the job]"
    # Execute a command
    Test-Path "\$name"
    # Just wait for a bit...
    Start-Sleep 5
  }

  # Show the loop variable here is correct
  Write-Host "processing $_..."

  # pass the loop variable across the job-context barrier
  Start-Job $ScriptBlock -ArgumentList $_
}

# Wait for all to complete
While (Get-Job -State "Running") { Start-Sleep 2 }

# Display output from all jobs
Get-Job | Receive-Job

# Cleanup
Remove-Job *

(I generally like to provide a reference to the PowerShell documentation as supporting evidence but, alas, my search has been fruitless. If you happen to know where context separation is documented, post a comment here to let me know!)

Answer 4

To complete previous answers, you can also use Wait-Job to wait for all jobs to complete:

For ($i=1; $i -le 3; $i++) {
    $ScriptBlock = {
        Param (
            [string] [Parameter(Mandatory=$true)] $increment
        )

        Write-Host $increment
    }

    Start-Job $ScriptBlock -ArgumentList $i
}

Get-Job | Wait-Job | Receive-Job

Answer 5

You can execute parallel jobs in Powershell 2 using Background Jobs. Check out Start-Job and the other job cmdlets.

# Loop through the server list
Get-Content "ServerList.txt" | %{

  # Define what each job does
  $ScriptBlock = {
    param($pipelinePassIn) 
    Test-Path "\\$pipelinePassIn\c`$\Something"
    Start-Sleep 60
  }

  # Execute the jobs in parallel
  Start-Job $ScriptBlock -ArgumentList $_
}

Get-Job

# Wait for it all to complete
While (Get-Job -State "Running")
{
  Start-Sleep 10
}

# Getting the information back from the jobs
Get-Job | Receive-Job