Python: os.read() blocking on fd

问题:

How to avoid the problem of line buffering if the fd is a bash shell?

回答1:

Here's a summary of what I've understood from your question and comments plus an unsatisfactory workaround at the end of the answer. The summary should help others to answer your question.

It seems you have a program (non-python) that wants to execute arbitrary shell commands that might require a tty (Q: Why not just use a pipe (popen())?) and you have chosen as a solution to execute a Python script that runs pty.spawn("/bin/sh", read_callback) and write/read to its stdin/stdout.

And now you have a problem that you can't find the end of output for a shell command i.e., you don't know when to stop reading program.py stdout and if you try to read too much then your non-python program blocks.

First of all, os.read inside read_callback does not block. It may return less than 1024 bytes but it doesn't block:

def read_callback(fd):
    data = os.read(fd, 1024) # <-- this doesn't block
    return data 

Though it doesn't help when the parent program tries to read.

Workaround

To avoid blocking on read, the parent can change PS1 environment in the running sh to some unique value or inject echo something_unique after each command. Then it can read one byte at a time until it reads something_unique. It should stop at this point.

As an alternative you could try to make the pipes nonblocking in your parent and read more than one byte at a time. Or use an exchange protocol that defines clear message boundaries and run shell commands one by one e.g., using pexpect.run() (to detect the end of output easily in the Python side).

回答2:

This seems to work:

#!/usr/local/cpython-3.3/bin/python

import os
import pty
#mport sys
import fcntl

OFLAGS = None

def set_nonblocking(file_handle):
    """Make a file_handle non-blocking."""
    global OFLAGS
    OFLAGS = fcntl.fcntl(file_handle, fcntl.F_GETFL)
    nflags = OFLAGS | os.O_NONBLOCK
    fcntl.fcntl(file_handle, fcntl.F_SETFL, nflags)


def main():
    (pid, file_handle) = pty.fork()
    if pid == 0:
        # we're in the child
        os.execl('/bin/sh', '/bin/sh', '-i')
    else:
        #file_handle = os.open('/dev/null', os.O_RDONLY)
        set_nonblocking(file_handle)
        while True:
            try:
                # return 1-n bytes or exception if no bytes
                data = os.read(file_handle, 1024)
            except BlockingIOError:
                #sys.stdout.write('no data read\r')
                pass
            else:
                print(len(data), data)

main()

Sometimes the way to deal with nonblocking I/O, is to use a thread or subprocess, BTW. Then one thread or subprocess can block while others merrily do their work.

HTH