Python: Using popen poll on background process

Question

I am running a long process (actually another python script) in the background. I need to know when it has finished. I have found that Popen.poll() always returns 0 for a background process. Is there another way to do this?

p = subprocess.Popen("sleep 30 &", shell=True,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
a = p.poll()
print(a)

Above code never prints None.

Answer 1

You don't need to use the shell backgrounding & syntax, as subprocess will run the process in the background by itself

Just run the command normally, then wait until Popen.poll returns not None

import time
import subprocess

p = subprocess.Popen("sleep 30", shell=True)
# Better: p = subprocess.Popen(["sleep", "30"])

# Wait until process terminates
while p.poll() is None:
    time.sleep(0.5)

# It's done
print "Process ended, ret code:", p.returncode

Answer 2

You shouldn't run your script with ampersand at the end. Because shell forks your process and returns 0 exit code.

Answer 3

I think you want either the popen.wait() or popen.communicate() commands. Communicate will grab the stdout and stderr data which you've put into PIPE. If the other item is a Python script I would avoid running a shell=True call by doing something like:

p = subprocess.Popen([python.call, "my", params, (go, here)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate()
print(stdout)
print(stderr)

Of course these hold the main thread and wait for the other process to complete, which might be bad. If you want to busy wait then you could simply wrap your original code in a loop. (Your original code did print "None" for me, btw)

Example of the wrapping in a loop solution:

p = subprocess.Popen([python.call, "my", params, (go, here)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while p.poll() == None:
    # We can do other things here while we wait
    time.sleep(.5)
    p.poll()
(results, errors) = p.communicate()
if errors == '':
    return results
else:
    raise My_Exception(errors)