Paramiko recv()/read()/readline(s)() on stderr ret

I'm using paramiko to collect some information on a remote host and experience issues, when reading (read()/readline()/readlines()) from the stderr channel.

Sometimes stderr.read() returns an empty string which to me looks like a result of a race condition. However, according to documentation and examples I found on the internet, this seems the exact way to go.

I also tried to open a dedicated channel and make use of chan.recv_ready() / chan.recv_stderr_ready() and reading from respective channels in a loop via chan.recv() / chan.recv_stderr() which however results in the same behaviour.

Here's a minimal test case which - in my setup - reliably causes that behaviour.

import paramiko

class SSH:
    def __init__(self):
        self.ssh = paramiko.SSHClient()
        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.ssh.connect('127.0.0.1', port=31337, username='root', password='root')

        self.stdout = ''
        self.stderr = ''
        self.exit_code = 0

    def _run_cmd(self, cmd):
        self.stdout = ''
        self.stderr = ''

        stdin, stdout, stderr = self.ssh.exec_command(cmd)

        self.stdout = stdout.read()
        self.stderr = stderr.read()

        while not stdout.channel.exit_status_ready():
            pass
        self.exit_code = stdout.channel.recv_exit_status()

        if self.exit_code:
            print("ERROR: " + self.stderr)

    def process_list(self):
        self._run_cmd('ls /proc/ | grep -E "^[0-9]+$" | grep -v $$')
        lines = self.stdout.split('\n')[:-1]
        data = []

        for process in lines:
            process_data = {}
            process_data['pid'] = int(process)

            # fetching and parsing process status information from /proc/[PID]/status

            self._run_cmd('cat /proc/%d/status' % (int(process)))

            data.append(self.stdout)

        return data


data = SSH()
while True:
    print data.process_list()

What I get after a couple of runs (if not the first one already) is: ERROR: while I am expecting: ERROR: cat: /proc/12883/status: No such file or directory

How can I make sure stderr is ready to read from / I read all data on stderr?

long story short: I've run into most of these problems and came up with my ultimate solution to most of my ssh related problems. Therefore feel free to have a look at this implementation of exec_command avoiding most of the empty_response/stalling scenarios.

long story

The main problem here is that you exec_command() is non-blocking and spawns a thread that takes care of the channels communication. This thread is waiting for incoming data and puts it into the channels buffers while the main thread may go on. This is where your problem lies. You're reading your buffers too early, even before you checked that your side received the exit_status. Receiving the exit_status confirms that the remote process exited with the given status code. Receiving the remote status code is not an indication that you've received all the data that may still be in transit. It may even arrive out-of-order at your side, saying that the status_code may arrive even before all the data (stderr,stdout) was received.

While data is arriving, the main thread continues. In your case, the main thread tries to read from stdout and stderr once and then blocks until the exit_status is ready. Note that the channel thread may still receive data from your remote command invocation. Also note that you'll have to manually empty buffers once your side received the remote exit_status.

In your specific case this is what happens:

execute_command spawns a new thread (lets call id exec_thread, managing the remote command invocation
while data may be received by the recently spawned thread, the main thread goes on.
[main-thread] you're storing the current contents of the stdout,stderr buffers in self.stdout,self.stderr. Note that you do not know if you've received all of the data of stderr or stdout. Most likely you've just received a few chunks.
[exec_thread] happily receives data for stdout,stderr while [main-thread] goes on.
[main-thread] busy-blocks until the exit_status is received. Again note, that at this point your buffers may still be full with recently received stderr,stdout chunks.
[main-thread] the exit_status is stored in self.exit_code. [exec_thread] is still alive, may still receive some out of order data. input buffers might still be filled.
[main-thread] returns from run_cmd

Note that [exec_thread] for that channel - paramiko creates one thread per channel invocation (i.e. exec_command) - is still alive and they'll sum up idling around and creating problems until you manually close them (stdout.channel.close()).