Powershell parallel jobs output file

Question

I have already done a lot of research, but still can't figure out how to accomplish what I want to do.

I want to perform the same tasks parallel on 100 Linux servers.

Here is a simplified example of my script:

$computer=Get-Content "serverList.txt"
$jobArray=@()
$script={
    $cpuThresh=70
    $cpuUsage=<Get CPU usage of the host>
    Write-Host "CPU Usage: $cpuUsage %"
    if ($cpuUsage -ge $cpuThresh) {
        Write-Host "Unexpected CPU Usage" -ForegroundColor Red
    }
}
foreach ($system in $computer) {
    $jobArray += Start-Job -ScriptBlock $script -ArgumentList $system
    While ((Get-Job -State 'Running').Count -ge 10) {
        Start-Sleep -Milliseconds 10
    }
}
foreach ($job in $jobArray) {
    While ($job.State -eq 'Running') {
        Start-Sleep -Milliseconds 10
    }
    Receive-Job -Job $job
    Remove-Job -Job $job
}

The problem I have is that I want to write certain messages (e.g. Unexpected CPU Usage) to a separate file and multiple jobs are trying to write to this file at the same time.

My idea would be to save all messages into an array and write the content at the end of the script (second foreach loop) to a file.

But Receive-Job doesn't return any variables/objects.

Is there a way to return a variable/object? Or is there another way to achieve what I want to do?

I would appreciate any help. Thanks.

Answer 1

Every job has at least (an normally only one) child jobs. The output of the process is actually held in separate output buffers of the child jobs, and can be accessed from there. You can use Write-Verbose for one set of output, and Write-Warning for another, and read it back from the Verbose and Warning streams separately:

$computer=Get-Content "serverList.txt"
$jobArray=@()
$script={
    $VerbosePreference = 'Continue'
    $Args[0]
    $cpuThresh=70
    $cpuUsage=<Get CPU usage of the host>
    Write-Verbose "CPU Usage: $cpuUsage %"
    if ($cpuUsage -ge $cpuThresh) {
        Write-Warning "Unexpected CPU Usage" 
    }
$Results  = @{}
$Warnings = @{}
$Outputs  = @{}
}
foreach ($system in $computer) {
    $jobArray += Start-Job -ScriptBlock $script -ArgumentList $system
    While ((Get-Job -State 'Running').Count -ge 10) {
        Start-Sleep -Milliseconds 10
    }
}
foreach ($job in $jobArray) {
    While ($job.State -eq 'Running') {
        Start-Sleep -Milliseconds 10
    }
     $Server = $Job.ChildJobs[0].Output[0]
     $Results[$Server] = $Job.ChildJobs[0].Verbose
     $Warnings[$Server] = $Job.ChildJobs[0].Warning
     $Outputs[$Server]   = $Job.ChildJobs[0].Output

    Remove-Job -Job $job
}

Edit: updated for all local jobs.

Answer 2

Receive-Job does not get any results because Write-Host is used which is not a standard output. Replace the line Write-Host "Unexpected CPU Usage" -ForegroundColor Red with "Unexpected CPU Usage" and Receive-Job should start to receive the messages. Use Write-Host -ForegroundColor Red in the very end of your script when processing Receive-Job.

Also, I would recommend to take a look at the module SplitPipeline which is specifically designed for such tasks. Your script can use the command Split-Pipeline and its code will be reduced to minimum:

Get-Content "serverList.txt" | Split-Pipeline -Count 10 {process{
    $cpuThresh=70
    $cpuUsage = ... # Get CPU usage of the host, use $_ as the current input server
    "CPU Usage: $cpuUsage %" # standard output
    if ($cpuUsage -ge $cpuThresh) {
        "Unexpected CPU Usage" # "warning" to be analysed later
        # or even better, Split-Pipeline takes care of warnings:
        Write-Warning "Unexpected CPU Usage"
    }
}} | % {
    # process output here, e.g. normal messages goes to a log file
    # and warnings are processed as
    Write-Host "Unexpected CPU Usage" -ForegroundColor Red

    # or if you used Write-Warning above this is not even needed
}