I use a lot of os.system
calls to create background processes inside a for loop. How can I wait for all the background processes to end ?
os.wait
tells me there are no child process.
ps: I am using Solaris
here is my code :
#!/usr/bin/python
import subprocess
import os
pids = []
NB_PROC=30
for i in xrange(NB_PROC):
p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
pids.insert(0,p)
p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
pids.insert(0,p)
for i in xrange(NB_PROC*2):
pids[i].wait()
os.system("rm test.php*")
Normally, os.system()
returns when the child process is finished. So there is indeed nothing for os.wait()
to do. It is equivalent to subprocess.call()
.
Use subprocess.Popen()
to create background processes, and then the wait()
or poll()
methods of the Popen
objects to wait for them to quit.
By default, Popen does not spawn a shell but executes the program directly. This saves resources and prevents possible shell injection attacks.
According to the documentation for os.system()
:
The subprocess module provides more powerful facilities for spawning
new processes and retrieving their results; using that module is
preferable to using this function
If you want to do multiple jobs in parallel, consider using multiprocessing
, especially the Pool
object. It takes care of a lot of the details of farming work out over several processes.
Edit: Timing the execution of a program;
import time
import subprocess
t1 = time.clock()
t2 = time.clock()
overhead = t2-t1
t1 = time.clock()
subprocess.call(['wget', 'http://site.com/test.php'])
t2 = time.clock()
print 'elapsed time: {:.3f} seconds.'.format(t2-t1-overhead)
the solution was indeed in the subprocess module
#!/usr/bin/python
import subprocess
import os
pids = []
NB_PROC=4
cmd="(time wget http://site.com/test.php 2>&1 | grep elapsed | cut -d ' ' -f 3)"
for i in xrange(NB_PROC):
p = subprocess.Popen(cmd,stdin=None,stdout=None, shell=True)
pids.insert(0,p)
print "request %d processed" % (i+1)
for i in xrange(NB_PROC):
pids[i].wait()
os.system("rm test.php*")
switched to debian in the process but for some reason sometimes the scripts hangs while sometimes it just runs fine