Live stream stdout and stdin with websocket

Question

online compiler this is my website where users can run console programs.

At present, the user has to enter the program input before running the program. I am trying to build live user input for the program(want to give the same experience as they run programs on their laptop).

In research to achieve this I came across a solution to stream stdout and stdin with websocket.

My implementation

# coding: utf-8
import subprocess
import thread

from tornado.websocket import WebSocketHandler

from nbstreamreader import NonBlockingStreamReader as NBSR


class WSHandler(WebSocketHandler):
    def open(self):
        self.write_message("connected")
        self.app = subprocess.Popen(['sh', 'app/shell.sh'], stdout=subprocess.PIPE, stdin=subprocess.PIPE,
                                    shell=False)
        self.nbsr = NBSR(self.app.stdout)
        thread.start_new_thread(self.soutput, ())

    def on_message(self, incoming):
        self.app.stdin.write(incoming)

    def on_close(self):
        self.write_message("disconnected")

    def soutput(self):
        while True:
            output = self.nbsr.readline(0.1)
            # 0.1 secs to let the shell output the result
            if not output:
                print 'No more data'
                break
            self.write_message(output)

nbstreamreader.py

from threading import Thread
from Queue import Queue, Empty


class NonBlockingStreamReader:
    def __init__(self, stream):
        '''
        stream: the stream to read from.
                Usually a process' stdout or stderr.
        '''

        self._s = stream
        self._q = Queue()

        def _populateQueue(stream, queue):
            '''
            Collect lines from 'stream' and put them in 'quque'.
            '''

            while True:
                line = stream.readline()
                if line:
                    queue.put(line)
                else:
                    raise UnexpectedEndOfStream

        self._t = Thread(target=_populateQueue,
                         args=(self._s, self._q))
        self._t.daemon = True
        self._t.start()  # start collecting lines from the stream

    def readline(self, timeout=None):
        try:
            return self._q.get(block=timeout is not None,
                               timeout=timeout)
        except Empty:
            return None


class UnexpectedEndOfStream(Exception): pass

shell.sh

#!/usr/bin/env bash
echo "hello world"
echo "hello world"
read -p "Your first name: " fname
read -p "Your last name: " lname
echo "Hello $fname $lname ! I am learning how to create shell scripts"

This code streams stdout un-till shell.sh code reaches read statement.

Please guide me what wrong I am doing. Why it doesn't wait for stdin and reaches print 'No more data' before complete program executions?

Source code to test it https://github.com/mryogesh/streamconsole.git

Answer 1

Your readline() method times out unless you send input within 100ms, which then breaks the loop. The reason you don't see the read -p prompt is buffering (because of readline and pipe buffering). Finally, your example javascript doesn't send a trailing newline, so read will not return.

If you increase the timeout, include a newline, and find a way to work around buffering issues, your example should basically work.

I'd also use tornado.process and coroutines instead of subprocess and thread:

from tornado import gen
from tornado.process import Subprocess
from tornado.ioloop import IOLoop
from tornado.iostream import StreamClosedError
from tornado.websocket import WebSocketHandler


class WSHandler(WebSocketHandler):
    def open(self):
        self.app = Subprocess(['script', '-q', 'sh', 'app/shell.sh'], stdout=Subprocess.STREAM, stdin=Subprocess.STREAM)
        IOLoop.current().spawn_callback(self.stream_output)

    def on_message(self, incoming):
        self.app.stdin.write(incoming.encode('utf-8'))

    @gen.coroutine
    def stream_output(self):
        try:
            while True:
                line = yield self.app.stdout.read_bytes(1000, partial=True)
                self.write_message(line.decode('utf-8'))
        except StreamClosedError:
            pass

Answer 2

Maybe you should take a look at websocketd (homepage).
I have a feeling it will simplify what at you want to do significantly.
It will act as websocket server and start program you give it as parameter every time client connects. Everything that comes over the websocket connection will be forwarded to STDIN of that program. Everything that program outputs to STDOUT will be sent over websocket.