Why does my Python thread with subprocess not work

2019-07-26 07:20发布

问题:

I have the following subtle problem: The Python code starts two threads, each creating a subprocess call to an exexutable (in fact written in C). The first executable is passed the argument 10000, meaning a delay of 10s before its exits. Similar for the second executable, but with a delay of 20s.

I observe, that worker1 and worker 2 do their printouts at the same time, namely after 20s, the longer of both delays. Why is wait() for worker1 somehow "blocked" by the longer delay of the other? Whatever I pass as arguments, both print outputs are right after the longer of both times.

Note: This is a boiled down example for demonstration purposes and not the code I'm working on. Anyway, it is not related to print output, in my original code instead of print I do something more complicated.

import subprocess
import threading
import time


def worker1():
    proc = subprocess.Popen(["../testapp1.exe", "10000"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    outs, errs = proc.communicate()
    ret = proc.wait()
    print("result worker1:%s ret:%d" % (outs, ret))
    print("done")


def worker2():
    proc = subprocess.Popen(["../testapp2.exe", "20000"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    outs, errs = proc.communicate()
    ret = proc.wait()
    print("result worker2:%s ret:%d" % (outs, ret))
    print("done")


job_thread1 = threading.Thread(target = worker1)
job_thread1.start()

job_thread2 = threading.Thread(target = worker2)
job_thread2.start()

job_thread1.join()
print ("after join1")
job_thread2.join()
print ("after join2")
# time.sleep(20000) original

The executable:

#include <string.h>
#include <windows.h>


int main(int argc, char *argv[])
{
  int count;
  //printf ("This program was called with \"%s\".\n",argv[0]);

  int sleep = 0;

  if (argc > 1)
  {
      for (count = 1; count < argc; count++)
      {
          if (count == 1)
          {
              sleep = atoi(argv[1]);
          }
          printf("\nargv[%d] = %s\n", count, argv[count]);
      }
  }
  else
  {
      printf("The command had no other arguments.\n");
  }

  int fin = sleep;
  for (int i=0; i<fin; i++)
  {
      for (int i2=0; i2<100; i2++);
  }
  //Sleep(sleep);

  return 0;
}

EDIT: when I remove "communicate" it works for some reason. But in this case I cannot get the output of the process...

def worker1():
    proc = subprocess.Popen(["../testapp1.exe", "10000"], shell=False, 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # outs, errs = proc.communicate()
    ret = proc.wait()
    print("result worker1:%s ret:%d" % (outs, ret))
    print("done")

EDIT2: Using only wait(), I found, that it is blocking at the marked label, where the output from the pipe is read...

def worker1():
    try:
        proc = subprocess.Popen(["../testapp.exe", "100000000"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        ret = proc.wait()
        xread = proc.stdout.read() ### !!!! BLOCKS
        print("xread:",xread)  
        sys.stdout.write("done-1")
        sys.stdout.flush()
    except ComError as e:
        print("got exception")
        raise e