My eventual goal is to redirect the stdout from several subprocesses to some queues, and print those out somewhere (maybe in a little GUI).
The first step is to subclass Queue into an object that behaves much like the stdout. But that is where I got stuck. Subclassing the multiprocessing Queue seems impossible in Python v3.5.
# This is a Queue that behaves like stdout
# Unfortunately, doesn't work in Python 3.5 :-(
class StdoutQueue(Queue):
def __init__(self,*args,**kwargs):
Queue.__init__(self,*args,**kwargs, ctx='')
def write(self,msg):
self.put(msg)
def flush(self):
sys.__stdout__.flush()
I found this snippet in the following post (probably Python 3.5 did not yet exist at that moment): Python multiprocessing redirect stdout of a child process to a Tkinter Text
In Python v3.5 you stumble on strange error messages when subclassing the multiprocessing Queue class. I found two bug reports describing the issue:
https://bugs.python.org/issue21367
https://bugs.python.org/issue19895
I have 2 questions:
- Suppose I want to stick to Python v3.5 - going to a previous version is not really an option. What workaround can I use to subclass the multiprocessing Queue somehow?
- Is the bug still around if I upgrade to Python v3.6?
EDIT :
There is a known issue when you try to subclass the Queue class found in here:
from multiprocessing import Queue # <- known issue: you cannot subclass
# this Queue class, because it is
# not a genuine python class.
But the following should work:
from multiprocessing.queues import Queue # <- from this Queue class, you
# should be able to make a
# subclass. But Python 3.5
# refuses :-(
Sadly, even that doesn't work in Python v3.5. You get the following error:
C:\Users\..\myFolder > python myTest.py
Traceback (most recent call last):
File "myTest.py", line 49, in <module>
q = StdoutQueue()
File "myTest.py", line 22, in __init__
super(StdoutQueue,self).__init__(*args,**kwargs)
TypeError: __init__() missing 1 required keyword-only argument: 'ctx'
EDIT :
Thank you Darth Kotik for solving the problem! Here is the complete code, updated with his solution. Now it works.
import sys
import time
import multiprocessing as mp
import multiprocessing.queues as mpq
from threading import Thread
from tkinter import *
'''-------------------------------------------------------------------'''
''' SUBCLASSING THE MULTIPROCESSING QUEUE '''
''' '''
''' ..and make it behave as a general stdout io '''
'''-------------------------------------------------------------------'''
# The StdoutQueue is a Queue that behaves like stdout.
# We will subclass the Queue class from the multiprocessing package
# and give it the typical stdout functions.
#
# (1) First issue
# Subclassing multiprocessing.Queue or multiprocessing.SimpleQueue
# will not work, because these classes are not genuine
# python classes.
# Therefore, you need to subclass multiprocessing.queues.Queue or
# multiprocessing.queues.SimpleQueue . This issue is known, and is not
# the reason for asking this question. But I mention it here, for
# completeness.
#
# (2) Second issue
# There is another problem that arises only in Python V5 (and beyond).
# When subclassing multiprocessing.queues.Queue, you have to provide
# a 'multiprocessing context'. Not doing that, leads to an obscure error
# message, which is in fact the main topic of this question. Darth Kotik
# solved it.
# His solution is visible in this code:
class StdoutQueue(mpq.Queue):
def __init__(self,*args,**kwargs):
ctx = mp.get_context()
super(StdoutQueue, self).__init__(*args, **kwargs, ctx=ctx)
def write(self,msg):
self.put(msg)
def flush(self):
sys.__stdout__.flush()
'''-------------------------------------------------------------------'''
''' TEST SETUP '''
'''-------------------------------------------------------------------'''
# This function takes the text widget and a queue as inputs.
# It functions by waiting on new data entering the queue, when it
# finds new data it will insert it into the text widget.
def text_catcher(text_widget,queue):
while True:
text_widget.insert(END, queue.get())
def test_child(q):
# This line only redirects stdout inside the current process
sys.stdout = q
# or sys.stdout = sys.__stdout__ if you want to print the child to the terminal
print('child running')
def test_parent(q):
# Again this only redirects inside the current (main) process
# commenting this like out will cause only the child to write to the widget
sys.stdout = q
print('parent running')
time.sleep(0.5)
mp.Process(target=test_child,args=(q,)).start()
if __name__ == '__main__':
gui_root = Tk()
gui_txt = Text(gui_root)
gui_txt.pack()
q = StdoutQueue()
gui_btn = Button(gui_root, text='Test', command=lambda:test_parent(q),)
gui_btn.pack()
# Instantiate and start the text monitor
monitor = Thread(target=text_catcher,args=(gui_txt,q))
monitor.daemon = True
monitor.start()
gui_root.mainloop()
