python multiprocessing/threading code exits early

2019-07-07 20:10发布

问题:

I'm trying to create multiple processes which each call multiple threads. I'm running the following code with python3.5

A simplified example of the problem looks like this:

import multiprocessing
import time
import threading    

class dumb(threading.Thread):
    def __init__(self):
        super(dumb, self).__init__()
    def run(self):
        while True:
            print("hi")
            time.sleep(1)    

def test():
    for i in range(2):
        bar = dumb()
        bar.start()
def main():
    p = []
    for i in range(2):
        p.append(multiprocessing.Process(target=test))
    for i in p:
        i.start()
    for i in p:
        i.join()
if __name__ == '__main__':
    main()

I expect this code to print "hi" forever, but it prints exactly once for each thread in each process (4 times total).

When I remove the multiprocessing, the multi-threading works.

When I remove the multi-threading, the multiprocessing works.

The part that I think is the problem after I read the multiprocessing documentation: The documentation for join states: "Block the calling thread until the process whose join() method is called terminates or until the optional timeout occurs."

If this was working as I expect it to, main would wait forever when trying to join.

I put try/except blocks around the while loop and didn't see any errors.

I have tried passing the "dumb" class a queue and passing exceptions through, but the queue stays empty.

Any tips on how to debug this would be greatly appreciated. My best guesses are

  • The thread is exiting early (print statements after while loop were never hit though)

  • Main exits and the processes are killed

  • Join is working properly, but not the way I expect it to?

Any ideas?

回答1:

Resolution: the new bug was closed as a duplicate of http://bugs.python.org/issue18966

Alas, there's no simple, satisfying explanation "for why". The cause is that multiprocessing arranges for worker processes to leave Python via calling os._exit() rather than the normal sys.exit(). os._exit() skips all "normal" shutdown processing. Part of what's skipped is .join()-ing non-daemon threads, so the process just vanishes while the threads are still running.

That should at least (according to me) be documented, or preferably changed.

In the meantime, the workaround - as you already know - is to explicitly .join() the threads yourself.

ANOTHER WAY

Under Python 3.4 or later, you could also use multiprocessing's spawn start method:

https://docs.python.org/3/library/multiprocessing.html?highlight=spawn#contexts-and-start-methods

That causes worker processes to finish via sys.exit(exitcode), which does all the normal shutdown processing (including .join()-ing non-daemon threads).

spawn is the only start method available on Windows, which explains why I had no problem running your original example.