multithreading issue with wx.TextCtrl (or underlyi

Question

I am developing a GUI to launch an external long-term running background program. This background program can be given input command via stdin and use stdout and stderr to keep printing out output and error messages. I use a wx.TextCtrl object inside the GUI to give input and print output. My current code is as follows, which is mainly inspired by the "how to implemente a shell GUI window" post: wxPython: how to create a bash shell window?

However, my following code uses "buffer previous output" approach, i.e., I use a thread to buffer the output. The buffered transaction output could be only rendered when I give next input command and press "return" button. Now, I want to see the output messages timely, therefore I want to have the feature that "the output can always be spontaneously printed out (directly flushed out) from the background subprocess and I could also interfere to type some input command via stdin and have the output printed.

class BashProcessThread(threading.Thread):
    def __init__(self, readlineFunc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.lines = []
        self.outputQueue = Queue.Queue()
        self.setDaemon(True)

    def run(self):
        while True:
           line = self.readlineFunc()
           self.outputQueue.put(line)
           if (line==""):
            break
        return ''.join(self.lines)

    def getOutput(self):
        """ called from other thread """            
        while True:
            try:
                line = self.outputQueue.get_nowait()
                lines.append(line)
            except Queue.Empty:
                break
        return ''.join(self.lines)

class myFrame(wx.Frame):
    def __init__(self, parent, externapp):
        wx.Window.__init__(self, parent, -1, pos=wx.DefaultPosition)
        self.textctrl = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
        launchcmd=["EXTERNAL_PROGRAM_EXE"]
        p = subprocess.Popen(launchcmd, stdin=subprocess.PIPE, 
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        self.outputThread = BashProcessThread(p.stdout.readline)
        self.outputThread.start()
        self.__bind_events()         
        self.Fit()

    def __bind_events(self):
        self.Bind(wx.EVT_TEXT_ENTER, self.__enter)

    def __enter(self, e):
        nl=self.textctrl.GetNumberOfLines()
        ln =  self.textctrl.GetLineText(nl-1)
        ln = ln[len(self.prompt):]     
        self.externapp.sub_process.stdin.write(ln+"\n")
        time.sleep(.3)
        self.textctrl.AppendText(self.outputThread.getOutput())

How should I modify the above code to achieve this? Do I still need to use thread? Can I code a thread as follows?

class PrintThread(threading.Thread):
    def __init__(self, readlineFunc, tc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.textctrl=tc
        self.setDaemon(True)

    def run(self):
        while True:
            line = self.readlineFunc()
            self.textctrl.AppendText(line)

However, when I tried with the above code, it crashes.

I have errors from Gtk, as follows.

(python:13688): Gtk-CRITICAL **: gtk_text_layout_real_invalidate: assertion `layout->wrap_loop_count == 0' failed
Segmentation fault

or sometimes the error

 (python:20766): Gtk-CRITICAL **: gtk_text_buffer_get_iter_at_mark: assertion `GTK_IS_TEXT_MARK (mark)' failed
Segmentation fault

or sometimes the error

(python:21257): Gtk-WARNING **: Invalid text buffer iterator: either the iterator is uninitialized, or the characters/pixbufs/widgets in the buffer have been modified since the iterator was created.
You must use marks, character numbers, or line numbers to preserve a position across buffer modifications.
You can apply tags and insert marks without invalidating your iterators,
but any mutation that affects 'indexable' buffer contents (contents that can be referred to by character offset)
will invalidate all outstanding iterators
Segmentation fault

or sometimes the error

Gtk-ERROR **: file gtktextlayout.c: line 1113 (get_style): assertion failed: (layout->one_style_cache == NULL)
aborting...
Aborted

or other error message, but different error message each time, really weird!

It seems the wx.TextCtrl or the underlying gui control of GTK+ has some problem with multi-threading. sometimes I do not type any input commands, it also crashes. I search from the certain post on the internet, and looks like it is dangerous to call a GUI control from a secondary thread.

I found my mistake. As pointed out in wxpython -- threads and window events or in chapter 18 of the book "WxPython in action" by Noel and Robin:

The most important point is that GUI operations must take place in the main thread, or the one that the application loop is running in. Running GUI operations in a separate thread is a good way for your application to crash in unpredictable and hard-to-debug ways...

My mistake is I tried to pass the wx.TextCtrl object to another thread. This is not right. I will reconsider my design.

Answer 1

Cant you just pass the TextCtrl object to your OutputThread and have it directly append text to the output without tying it with the __enter method?

Answer 2

pty.spawn() could be useful. Also you can manually create PTYs with pty.openpty() and pass them to popen as stdin/stdout.

If you have access to text-mode program source, you can also disable buffering there.

Answer 3

The calling contract for anything in gtk (upon which wxpython is built and the thing which is failing the assertion) application is that you must only modify the gui controls from the Main Gui thread. So the answer for me is simple:

In C#, I needed to do the following (which is an anonymous lambda function, and there's almost certainly a similar library call in wxpython/gtk)

Gtk.Application.Invoke((_,__) =>
{
    //code which can safely modify the gui goes here
});

And that should sort out your assertion problems... it did for me, at least.

Answer 4

The biggest problem is that you can't force the subprocess not to buffer its output, and the standard I/O library of most programs will buffer the output when stdout is a pipe (more accurately, they will go from line-buffering to block buffering). Tools like Expect fix this by running the subprocess in a pseudo-tty, which basically tricks the subprocess into thinking its output is going to a terminal.

There is a Python module called Pexpect, which solves this in the same way that Expect does. I've never used it, so caveat emptor.