Python custom delimiter for read or readline

Question

I am interacting with a subprocess and trying to detect when it is ready for my input. The problem that I am having is that the read or readline functions rely on the '\n' delimiter at the end of the line, or an EOF to yield. Since this subprocess never exits, there is no EOF in the file like object. Since the keyword that I want to trigger off of does not contain that delimiter the read and readline functions never yield. For example:

'Doing something\n'
'Doing something else\n'
'input>'

Since this process never exits, the read or read line never see an EOF or \n that it requires to yield.

Is there a way to read this file like object and to set a custom delimiter to input>?

Answer 1

You can implement your own readlines function and choose the delimiter yourself:

def custom_readlines(handle, line_separator="\n", chunk_size=64):
    buf = ""  # storage buffer
    while not handle.closed:  # while our handle is open
        data = handle.read(chunk_size)  # read `chunk_size` sized data from the passed handle
        if not data:  # no more data...
            break  # break away...
        buf += data  # add the collected data to the internal buffer
        if line_separator in buf:  # we've encountered a separator
            chunks = buf.split(line_separator)
            buf = chunks.pop()  # keep the last entry in our buffer
            for chunk in chunks:  # yield the rest
                yield chunk + line_separator
    if buf:
        yield buf  # return the last buffer if any

Unfortunately, due to Python default buffering policies you won't be able to grab large swaths of data if they are not provided by the process you're calling, but you can always resort to setting the chunk_size to 1 and then read the input character by character. So, for your example, all you need to do is:

import subprocess

proc = subprocess.Popen(["your", "subprocess", "command"], stdout=subprocess.PIPE)

while chunk in custom_readlines(proc.stdout, ">", 1):
    print(chunk)
    # do whatever you want here...

And it should capture everything up to > from your subprocesses' STDOUT. You can also use multiple characters as separators in this version.