Multiprocessing code works upon import, breaks upo

Question

In a file called test.py I have

print 'i am cow'
import multi4
print 'i am cowboy'

and in multi4.py I have

import multiprocessing as mp
manager = mp.Manager()
print manager

I am confused by the way this code operates.

At the command line, if I type python and then in the python environment if I type import test.py I get the expected behavior:

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

>>>import test
i am cow
<multiprocessing.managers.SyncManager object at 0x025209B0>
i am cowboy 
>>>

However if I type test.py at the command line, I get

i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow
i am cow

Which would presumably go on forever unless I kill it. When I kill it, I get a bunch of repeated errors:

KeyboardInterrupt
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 373, in main
    prepare(preparation_data)
  File "C:\Python27\lib\multiprocessing\forking.py", line 488, in prepare
    '__parents_main__', file, path_name, etc
KeyboardInterrupt
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 373, in main
    prepare(preparation_data)
  File "C:\Python27\lib\multiprocessing\forking.py", line 488, in prepare
    '__parents_main__', file, path_name, etc
KeyboardInterrupt

So what is going on? Why does it behave one way under import, and another when I try to run it?

Answer 1

multiprocessing won't work properly in the interactive prompt on Windows, because it can't properly re-import __main__ in the child processes it spawns. However, that actually actually helps you here, since it keeps the manager = mp.Manager() line from being recursively executed in the child process that spawns when the Manager starts up.

In the actual script, however, the child can re-import __main__ properly. You're seeing the infinite recursion because you're not protecting the call to mp.Manager() with an if __name__ == "__main__": guard, which is required on Windows to prevent mp.Manager() from being executed in the child when the re-import occurs:

import multiprocessing as mp
if __name__ == "__main__":
    manager = mp.Manager()
    print manager

Edit:

Your example, where your main script (test.py) imports the module that creates the Manager, needs a bit of refactoring. You need to instantiate the Manager from the main script, by calling a method in the module that actually uses multiprocessing:

print 'i am cow'
import multi4
if __name__ == "__main__":
    multi4.init_manager()
print 'i am cowboy'

multi4.py

import multiprocessing as mp
manager = None
def init_manager():
    global manager
    manager = mp.Manager()

This is the only way to make sure you only create the Manager when you're actually executing a script.