Python, Popen and select - waiting for a process t

Question

I run a subprocess using:

  p = subprocess.Popen("subprocess", 
                       stdout=subprocess.PIPE, 
                       stderr=subprocess.PIPE, 
                       stdin=subprocess.PIPE)

This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.

I tried this:

  SECONDS_TO_WAIT = 10
  select.select([], 
                [p.stdout, p.stderr], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

but it just returns:

  ([],[],[])

on either condition. What can I do?

Answer 1

This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.

def runCmd(cmd, timeout=None):
    '''
    Will execute a command, read the output and return it back.

    @param cmd: command to execute
    @param timeout: process timeout in seconds
    @return: a tuple of three: first stdout, then stderr, then exit code
    @raise OSError: on missing command or if a timeout was reached
    '''

    ph_out = None # process output
    ph_err = None # stderr
    ph_ret = None # return code

    p = subprocess.Popen(cmd, shell=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)
    # if timeout is not set wait for process to complete
    if not timeout:
        ph_ret = p.wait()
    else:
        fin_time = time.time() + timeout
        while p.poll() == None and fin_time > time.time():
            time.sleep(1)

        # if timeout reached, raise an exception
        if fin_time < time.time():

            # starting 2.6 subprocess has a kill() method which is preferable
            # p.kill()
            os.kill(p.pid, signal.SIGKILL)
            raise OSError("Process timeout has been reached")

        ph_ret = p.returncode


    ph_out, ph_err = p.communicate()

    return (ph_out, ph_err, ph_ret)

Answer 2

This is a paraphrase on Evan's answer, but it takes into account the following :

1. Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(

2. There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).

     DEVNULL = open(os.devnull, "wb")
     process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout
   
     def kill_process():
     """ Kill process helper"""
     try:
        process.kill()
      except OSError:
        pass  # Swallow the error
   
     timer = Timer(timeout_in_sec, kill_process)
     timer.start()
   
     process.wait()
     timer.cancel()
   

Answer 3

Have you tried using the Popen.Poll() method. You could just do this:

p = subprocess.Popen("subprocess", 
                   stdout=subprocess.PIPE, 
                   stderr=subprocess.PIPE, 
                   stdin=subprocess.PIPE)

time.sleep(SECONDS_TO_WAIT)
retcode = p.poll()
if retcode is not None:
   # process has terminated

This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.

---

Edit:

How about:

t_nought = time.time()
seconds_passed = 0

while(p.poll() is not None and seconds_passed < 10):
    seconds_passed = time.time() - t_nought

if seconds_passed >= 10:
   #TIMED OUT

This has the ugliness of being a busy wait, but I think it accomplishes what you want.

Additionally looking at the select call documentation again I think you may want to change it as follows:

SECONDS_TO_WAIT = 10
  select.select([p.stderr], 
                [], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).

I hope this helps.

Answer 4

Here is a nice example:

from threading import Timer
from subprocess import Popen, PIPE

def kill_proc():
    proc.kill()

proc = Popen("ping 127.0.0.1", shell=True)
t = Timer(60, kill_proc)
t.start()
proc.wait()

Answer 5

Python 3.3

import subprocess as sp

try:
    sp.check_call(["/subprocess"], timeout=10,
                  stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
except sp.TimeoutError:
    # timeout (the subprocess is killed at this point)
except sp.CalledProcessError:
    # subprocess failed before timeout
else:
    # subprocess ended successfully before timeout

See TimeoutExpired docs.

Answer 6

Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:

from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor    

def stop(r):
    reactor.stop()
def eb(reason):
    reason.printTraceback()
def cb(result):
    stdout, stderr, exitcode = result
    # do something
getProcessOutputAndValue('/bin/someproc', []
    ).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()

Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:

http://twistedmatrix.com/projects/core/documentation/howto/process.html