How to capture process output asynchronously in po

Question

I want to capture stdout and stderr from a process that I start in a Powershell script and display it asynchronously to the console. I've found some documentation on doing this through MSDN and other blogs.

After creating and running the example below, I can't seem to get any output to be displayed asynchronously. All of the output is only displayed when the process terminates.

$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "cmd.exe"
$ps.StartInfo.UseShellExecute = $false
$ps.StartInfo.RedirectStandardOutput = $true
$ps.StartInfo.Arguments = "/c echo `"hi`" `& timeout 5"

$action = { Write-Host $EventArgs.Data  }
Register-ObjectEvent -InputObject $ps -EventName OutputDataReceived -Action $action | Out-Null

$ps.start() | Out-Null
$ps.BeginOutputReadLine()
$ps.WaitForExit()

In this example, I was expecting to see the output of "hi" on the commandline before the end of program execution because the OutputDataReceived event should have been triggered.

I've tried this using other executables - java.exe, git.exe, etc. All of them have the same effect, so I'm left to think that there is something simple that I'm not understanding or have missed. What else needs to be done to read stdout asynchronously?

Answer 1

Unfortunately asynchronous reading is not that easy if you want to do it properly. If you call WaitForExit() without timeout you could use something like this function I wrote (based on C# code):

function Invoke-Executable {
    # Runs the specified executable and captures its exit code, stdout
    # and stderr.
    # Returns: custom object.
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [String]$sExeFile,
        [Parameter(Mandatory=$false)]
        [String[]]$cArgs,
        [Parameter(Mandatory=$false)]
        [String]$sVerb
    )

    # Setting process invocation parameters.
    $oPsi = New-Object -TypeName System.Diagnostics.ProcessStartInfo
    $oPsi.CreateNoWindow = $true
    $oPsi.UseShellExecute = $false
    $oPsi.RedirectStandardOutput = $true
    $oPsi.RedirectStandardError = $true
    $oPsi.FileName = $sExeFile
    if (! [String]::IsNullOrEmpty($cArgs)) {
        $oPsi.Arguments = $cArgs
    }
    if (! [String]::IsNullOrEmpty($sVerb)) {
        $oPsi.Verb = $sVerb
    }

    # Creating process object.
    $oProcess = New-Object -TypeName System.Diagnostics.Process
    $oProcess.StartInfo = $oPsi

    # Creating string builders to store stdout and stderr.
    $oStdOutBuilder = New-Object -TypeName System.Text.StringBuilder
    $oStdErrBuilder = New-Object -TypeName System.Text.StringBuilder

    # Adding event handers for stdout and stderr.
    $sScripBlock = {
        if (! [String]::IsNullOrEmpty($EventArgs.Data)) {
            $Event.MessageData.AppendLine($EventArgs.Data)
        }
    }
    $oStdOutEvent = Register-ObjectEvent -InputObject $oProcess `
        -Action $sScripBlock -EventName 'OutputDataReceived' `
        -MessageData $oStdOutBuilder
    $oStdErrEvent = Register-ObjectEvent -InputObject $oProcess `
        -Action $sScripBlock -EventName 'ErrorDataReceived' `
        -MessageData $oStdErrBuilder

    # Starting process.
    [Void]$oProcess.Start()
    $oProcess.BeginOutputReadLine()
    $oProcess.BeginErrorReadLine()
    [Void]$oProcess.WaitForExit()

    # Unregistering events to retrieve process output.
    Unregister-Event -SourceIdentifier $oStdOutEvent.Name
    Unregister-Event -SourceIdentifier $oStdErrEvent.Name

    $oResult = New-Object -TypeName PSObject -Property ([Ordered]@{
        "ExeFile"  = $sExeFile;
        "Args"     = $cArgs -join " ";
        "ExitCode" = $oProcess.ExitCode;
        "StdOut"   = $oStdOutBuilder.ToString().Trim();
        "StdErr"   = $oStdErrBuilder.ToString().Trim()
    })

    return $oResult
}

It captures stdout, stderr and exit code. Example usage:

$oResult = Invoke-Executable -sExeFile 'ping.exe' -cArgs @('8.8.8.8', '-a')
$oResult | Format-List -Force 

For more info and alternative implementations (in C#) read this blog post.

Answer 2

The examples here are all useful, but didn't completely suit my use case. I didn't want to invoke the command and exit. I wanted to open a command prompt, send input, read the output, and repeat. Here's my solution for that.

Create Utils.CmdManager.cs

using System;
using System.Diagnostics;
using System.Text;
using System.Threading;

namespace Utils
{
    public class CmdManager : IDisposable
    {
        const int DEFAULT_WAIT_CHECK_TIME = 100;
        const int DEFAULT_COMMAND_TIMEOUT = 3000;

        public int WaitTime { get; set; }
        public int CommandTimeout { get; set; }

        Process _process;
        StringBuilder output;

        public CmdManager() : this("cmd.exe", null, null) { }
        public CmdManager(string filename) : this(filename, null, null) { }
        public CmdManager(string filename, string arguments) : this(filename, arguments, null) { }

        public CmdManager(string filename, string arguments, string verb)
        {
            WaitTime = DEFAULT_WAIT_CHECK_TIME;
            CommandTimeout = DEFAULT_COMMAND_TIMEOUT;

            output = new StringBuilder();

            _process = new Process();
            _process.StartInfo.FileName = filename;
            _process.StartInfo.RedirectStandardInput = true;
            _process.StartInfo.RedirectStandardOutput = true;
            _process.StartInfo.RedirectStandardError = true;
            _process.StartInfo.CreateNoWindow = true;
            _process.StartInfo.UseShellExecute = false;
            _process.StartInfo.ErrorDialog = false;
            _process.StartInfo.Arguments = arguments != null ? arguments : null;
            _process.StartInfo.Verb = verb != null ? verb : null;

            _process.EnableRaisingEvents = true;
            _process.OutputDataReceived += (s, e) =>
            {
                lock (output)
                {
                    output.AppendLine(e.Data);
                };
            };
            _process.ErrorDataReceived += (s, e) =>
            {
                lock (output)
                {
                    output.AppendLine(e.Data);
                };
            };

            _process.Start();
            _process.BeginOutputReadLine();
            _process.BeginErrorReadLine();
            _process.StandardInput.AutoFlush = true;
        }

        public void RunCommand(string command)
        {
            _process.StandardInput.WriteLine(command);
        }

        public string GetOutput()
        {
            return GetOutput(null, CommandTimeout, WaitTime);
        }

        public string GetOutput(string endingOutput)
        {
            return GetOutput(endingOutput, CommandTimeout, WaitTime);
        }

        public string GetOutput(string endingOutput, int commandTimeout)
        {
            return GetOutput(endingOutput, commandTimeout, WaitTime);
        }

        public string GetOutput(string endingOutput, int commandTimeout, int waitTime)
        {
            string tempOutput = "";
            int tempOutputLength = 0;
            int amountOfTimeSlept = 0;

            // Loop until
            //  a) command timeout is reached
            //  b) some output is seen
            while (output.ToString() == "")
            {
                if (amountOfTimeSlept >= commandTimeout)
                {
                    break;
                }

                Thread.Sleep(waitTime);
                amountOfTimeSlept += waitTime;
            }

            // Loop until:
            //  a) command timeout is reached
            //  b) endingOutput is found
            //  c) OR endingOutput is null and there is no new output for at least waitTime
            while (amountOfTimeSlept < commandTimeout)
            {
                if (endingOutput != null && output.ToString().Contains(endingOutput))
                {
                    break;
                }
                else if(endingOutput == null && tempOutputLength == output.ToString().Length)
                {
                    break;
                }

                tempOutputLength = output.ToString().Length;

                Thread.Sleep(waitTime);
                amountOfTimeSlept += waitTime;
            }

            // Return the output and clear the buffer
            lock (output)
            {
                tempOutput = output.ToString();
                output.Clear();
                return tempOutput.TrimEnd();
            }
        }

        public void Dispose()
        {
            _process.Kill();
        }
    }
}

Then from PowerShell add the class and use it.

Add-Type -Path ".\Utils.CmdManager.cs"

$cmd = new-object Utils.CmdManager
$cmd.GetOutput() | Out-Null

$cmd.RunCommand("whoami")
$cmd.GetOutput()

$cmd.RunCommand("cd")
$cmd.GetOutput()

$cmd.RunCommand("dir")
$cmd.GetOutput()

$cmd.RunCommand("cd Desktop")
$cmd.GetOutput()

$cmd.RunCommand("cd")
$cmd.GetOutput()

$cmd.RunCommand("dir")
$cmd.GetOutput()

$cmd.Dispose()

Don't forget to call the Dispose() function at the end to clean up the process that is running in the background. Alternatively, you could close that process by running something like $cmd.RunCommand("exit")

Answer 3

Based on Alexander Obersht's answer I've created a function that uses timeout and asynchronous Task classes instead of event handlers. According to Mike Adelson

Unfortunately, this method(event handlers) provides no way to know when the last bit of data has been received. Because everything is asynchronous, it is possible (and I have observed this) for events to fire after WaitForExit() has returned.

function Invoke-Executable {
# from https://stackoverflow.com/a/24371479/52277
    # Runs the specified executable and captures its exit code, stdout
    # and stderr.
    # Returns: custom object.
# from http://www.codeducky.org/process-handling-net/ added timeout, using tasks
param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [String]$sExeFile,
        [Parameter(Mandatory=$false)]
        [String[]]$cArgs,
        [Parameter(Mandatory=$false)]
        [String]$sVerb,
        [Parameter(Mandatory=$false)]
        [Int]$TimeoutMilliseconds=1800000 #30min
    )
    Write-Host $sExeFile $cArgs

    # Setting process invocation parameters.
    $oPsi = New-Object -TypeName System.Diagnostics.ProcessStartInfo
    $oPsi.CreateNoWindow = $true
    $oPsi.UseShellExecute = $false
    $oPsi.RedirectStandardOutput = $true
    $oPsi.RedirectStandardError = $true
    $oPsi.FileName = $sExeFile
    if (! [String]::IsNullOrEmpty($cArgs)) {
        $oPsi.Arguments = $cArgs
    }
    if (! [String]::IsNullOrEmpty($sVerb)) {
        $oPsi.Verb = $sVerb
    }

    # Creating process object.
    $oProcess = New-Object -TypeName System.Diagnostics.Process
    $oProcess.StartInfo = $oPsi


    # Starting process.
    [Void]$oProcess.Start()
# Tasks used based on http://www.codeducky.org/process-handling-net/    
 $outTask = $oProcess.StandardOutput.ReadToEndAsync();
 $errTask = $oProcess.StandardError.ReadToEndAsync();
 $bRet=$oProcess.WaitForExit($TimeoutMilliseconds)
    if (-Not $bRet)
    {
     $oProcess.Kill();
    #  throw [System.TimeoutException] ($sExeFile + " was killed due to timeout after " + ($TimeoutMilliseconds/1000) + " sec ") 
    }
    $outText = $outTask.Result;
    $errText = $errTask.Result;
    if (-Not $bRet)
    {
        $errText =$errText + ($sExeFile + " was killed due to timeout after " + ($TimeoutMilliseconds/1000) + " sec ") 
    }
    $oResult = New-Object -TypeName PSObject -Property ([Ordered]@{
        "ExeFile"  = $sExeFile;
        "Args"     = $cArgs -join " ";
        "ExitCode" = $oProcess.ExitCode;
        "StdOut"   = $outText;
        "StdErr"   = $errText
    })

    return $oResult
}

Answer 4

I couldn't get either of these examples to work with PS 4.0.

I wanted to run puppet apply from an Octopus Deploy package (via Deploy.ps1) and see the output in "real time" rather than wait for the process to finish (an hour later), so I came up with the following:

# Deploy.ps1

$procTools = @"

using System;
using System.Diagnostics;

namespace Proc.Tools
{
  public static class exec
  {
    public static int runCommand(string executable, string args = "", string cwd = "", string verb = "runas") {

      //* Create your Process
      Process process = new Process();
      process.StartInfo.FileName = executable;
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = true;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;

      //* Optional process configuration
      if (!String.IsNullOrEmpty(args)) { process.StartInfo.Arguments = args; }
      if (!String.IsNullOrEmpty(cwd)) { process.StartInfo.WorkingDirectory = cwd; }
      if (!String.IsNullOrEmpty(verb)) { process.StartInfo.Verb = verb; }

      //* Set your output and error (asynchronous) handlers
      process.OutputDataReceived += new DataReceivedEventHandler(OutputHandler);
      process.ErrorDataReceived += new DataReceivedEventHandler(OutputHandler);

      //* Start process and handlers
      process.Start();
      process.BeginOutputReadLine();
      process.BeginErrorReadLine();
      process.WaitForExit();

      //* Return the commands exit code
      return process.ExitCode;
    }
    public static void OutputHandler(object sendingProcess, DataReceivedEventArgs outLine) {
      //* Do your stuff with the output (write to console/log/StringBuilder)
      Console.WriteLine(outLine.Data);
    }
  }
}
"@

Add-Type -TypeDefinition $procTools -Language CSharp

$puppetApplyRc = [Proc.Tools.exec]::runCommand("ruby", "-S -- puppet apply --test --color false ./manifests/site.pp", "C:\ProgramData\PuppetLabs\code\environments\production");

if ( $puppetApplyRc -eq 0 ) {
  Write-Host "The run succeeded with no changes or failures; the system was already in the desired state."
} elseif ( $puppetApplyRc -eq 1 ) {
  throw "The run failed; halt"
} elseif ( $puppetApplyRc -eq 2) {
  Write-Host "The run succeeded, and some resources were changed."
} elseif ( $puppetApplyRc -eq 4 ) {
  Write-Warning "WARNING: The run succeeded, and some resources failed."
} elseif ( $puppetApplyRc -eq 6 ) {
  Write-Warning "WARNING: The run succeeded, and included both changes and failures."
} else {
  throw "Un-recognised return code RC: $puppetApplyRc"
}

Credit goes to T30 and Stefan Goßner