Alternative to StreamReader.Peek and Thread.Interr

Question

Quick preface of what I'm trying to do. I want to start a process and start up two threads to monitor the stderr and stdin. Each thread chews off bits of the stream and then fires it out to a NetworkStream. If there is an error in either thread, both threads need to die immediately.

Each of these processes with stdout and stdin monitoring threads are spun off by a main server process. The reason this becomes tricky is because there can easily be 40 or 50 of these processes at any given time. Only during morning restart bursts are there ever more than 50 connections, but it really needs to be able to handle 100 or more. I test with 100 simultaneous connections.

try
{
    StreamReader reader = this.myProcess.StandardOutput;

    char[] buffer = new char[4096];
    byte[] data;
    int read;

    while (reader.Peek() > -1 ) // This can block before stream is streamed to
    {
        read = reader.Read(buffer, 0, 4096);
        data = Server.ClientEncoding.GetBytes(buffer, 0, read);
        this.clientStream.Write(data, 0, data.Length); //ClientStream is a NetworkStream
    }
}
catch (Exception err)
{
        Utilities.ConsoleOut(string.Format("StdOut err for client {0} -- {1}", this.clientID, err));
        this.ShutdownClient(true);
}

This code block is run in one Thread which is right now not Background. There is a similar thread for the StandardError stream. I am using this method instead of listening to OutputDataReceived and ErrorDataReceived because there was an issue in Mono that caused these events to not always fire properly and even though it appears to be fixed now I like that this method ensures I'm reading and writing everything sequentially.

ShutdownClient with True simply tries to kill both threads. Unfortunately the only way I have found to make this work is to use an interrupt on the stdErrThread and stdOutThread objects. Ideally peek would not block and I could just use a manual reset event to keep checking for new data on stdOut or stdIn and then just die when the event is flipped.

I doubt this is the best way to do it. Is there a way to execute this without using an Interrupt?

I'd like to change, because I just saw in my logs that I missed a ThreadInterruptException thrown inside Utlities.ConsoleOut. This just does a System.Console.Write if a static variable is true, but I guess this blocks somewhere.

Edits:

These threads are part of a parent Thread that is launched en masse by a server upon a request. Therefore I cannot set the StdOut and StdErr threads to background and kill the application. I could kill the parent thread from the main server, but this again would get sticky with Peek blocking.

Added info about this being a server.

Also I'm starting to realize a better Queuing method for queries might be the ultimate solution.

Answer 1

You're building a server. You want to avoid blocking. The obvious solution is to use the asynchronous APIs:

var myProcess = Process.GetCurrentProcess();
StreamReader reader = myProcess.StandardOutput;

char[] buffer = new char[4096];
byte[] data;
int read;

while (!myProcess.HasExited)
{
    read = await reader.ReadAsync(buffer, 0, 4096);
    data = Server.ClientEncoding.GetBytes(buffer, 0, read);

    await this.clientStream.WriteAsync(data, 0, data.Length);
}

No need to waste threads doing I/O work :)

Answer 2

Why dont you just set both Threads to be backround and then kill the app? It would cause an immediate closing of both threads.

Answer 3

Get rid of peek and use the method below to read from the process output streams. ReadLine() returns null when the process ends. To join this thread with your calling thread either wait for the process to end or kill the process yourself. ShutdownClient() should just Kill() the process which will cause the other thread reading the StdOut or StdErr to also exit.

    private void ReadToEnd()
    {
        string nextLine;
        while ((nextLine = stream.ReadLine()) != null)
        {
             output.WriteLine(nextLine);
        }
    }

Answer 4

I can tell this whole mess stems from the fact that Peek blocks. You're really trying to fix something that is fundamentally broken in the framework and that is never easy (i.e. not a dirty hack). Personally, I would fix the root of the problem, which is the blocking Peek. Mono would've followed Microsoft's implementation and thus ends up with the same problem.

While I know exactly how to fix the problem should I be allowed to change the framework source code, the workaround is lengthy and time consuming.

But here goes.

Essentially, what Microsoft needs to do is change Process.StartWithCreateProcess such that standardOutput and standardError are both assigned a specialised type of StreamReader (e.g. PipeStreamReader).

In this PipeStreamReader, they need to override both ReadBuffer overloads (i.e. need to change both overloads to virtual in StreamReader first) such that prior to a read, PeekNamedPipe is called to do the actual peek. As it is at the moment, FileStream.Read() (called by Peek()) will block on pipe reads when no data is available for read. While a FileStream.Read() with 0 bytes works well on files, it doesn't work all that well on pipes. In fact, the .NET team missed an important part of the pipe documentation - PeekNamedPipe WinAPI.

The PeekNamedPipe function is similar to the ReadFile function with the following exceptions:

...

The function always returns immediately in a single-threaded application, even if there is no data in the pipe. The wait mode of a named pipe handle (blocking or nonblocking) has no effect on the function.

The best thing at this moment without this issue solved in the framework would be to roll out your own Process class (a thin wrapper around WinAPI would suffice).