Using multiprocessing inside decorator generates e

Question

I came across a problem that I can't solve and it's associated with multiprocessing and use it inside the decorator.

When I'm calling the method run_in_parallels using multiprocessing I 'm getting the error:

Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase

The call takes place inside the decorator, then followed the above-mentioned problem. At the time of calling the same method run_in_parallels without a decorator all working properly.

What is the reason of this problem?

---

file: w_PythonHelper.py

desc: Function 'run_in_parallel' is used to run multiple processes simultaneously. The first method, which will end operation stops the others.

from multiprocessing import Process,Event

class ExtProcess(Process):
    def __init__(self, event,*args,**kwargs):
        self.event=event
        Process.__init__(self,*args,**kwargs)

    def run(self):
        Process.run(self)
        self.event.set()

class PythonHelper(object):
    @staticmethod
    def run_in_parallel(*functions):
        event=Event()
        processes=dict()
        for function in functions:
            fname=function[0]
            try:fargs=function[1]
            except:fargs=list()
            try:fproc=function[2]
            except:fproc=1
            for i in range(fproc):
                process=ExtProcess(event,target=fname,args=fargs)
                process.start()
                processes[process.pid]=process
        event.wait()
        for process in processes.values():
            process.terminate()
        for process in processes.values():
            process.join()

file: w_Recorder.py

desc: function 'capture' is used to grab a screenshot

from PIL import ImageGrab
import time

class Recorder(object):
    def capture(self):
        ImageGrab.grab().save("{f}.{e}".format(f=time.time(),e="png"))

file: w_Decorators.py

desc: Running parallel a given function along with a method 'capture' of class 'Recorder'

from w_Recorder import Recorder
from w_PythonHelper import PythonHelper

def check(function):
    def wrapper(*args):
        try:
            recorder=Recorder()
            PythonHelper.run_in_parallel([function,args],[recorder.capture])
            print("success")
        except Exception as e:
            print("failure: {}".format(e))
        return function
    return wrapper

file: w_Logger.py

desc: Main program (generates error)

from w_Decorators import check
import time

class Logger(object):

    @check
    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        self.run_testcase()


if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

file: w_Logger.py

desc: Main program (works corectly)

from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time

class Logger(object):

    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        recorder=Recorder()
        PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])

if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

What is the difference that these same methods presented in the two cases work differently?

---

EDIT: Does anyone has an idea to solve this problem (is this Python bug)? If not, maybe someone knows a good way to capture screen shots when the application is running?

Actually I found similiar question: multiprocessing.Process subclass works on Linux but not Windows

The answer is: To fix this, you can remove the process member., but how can I do this for my example.

While debugging the error occurs when calling process.start() in run_in_parallel(*functions)

---

EDIT2: like ivan_pozdeev wrote: i can use wrapper as a function, but can't use it as decorator. I have many functions decorated by this decorator and the simpliest way is to use multiprocessing inside decorator. But unfortunatly I can't solve this problem. Maybe someone has already solved a similar problem. I would be grateful for any hint.

'run_in_parallel' function works like I want. Two or more functions run in parallel and the first function, which is completed forces the termination of second function. When I call wrapper(function,*args) then functions works ok, when I put this mechanism inside decorator I get 'can't pickle function...it's not found as' error. Details can be found above

My Traceback:

Traceback (most recent call last):
  File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'run_testcase'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\EskyTests\w_Logger.py", line 19, in <module>
    logger.run_logger()
  File "C:\EskyTests\w_Logger.py", line 14, in run_logger
    self.run_testcase()
  File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
    PythonHelper.run_in_parallel([function,args],[recorder.capture])
  File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
    process.start()
  File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
    self.save(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
    save(state)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
    save(v)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
    (obj, module, name))
_pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase

Answer 1

Tricky, but I think what's happening is that check stores a reference to the unbound method when the class is defined. The working example uses a reference to the bound method self.run_testcase when you call run_logger.

I think the best thing is to try making run_testcase a top level function, rather than a method of a class.

Also, your capture function probably won't do what you expect - the current time will be stored when the function is defined, and every screenshot will be saved over the previous one. You probably want to call time.time() inside the function.

Answer 2

The function you're passing to Process.__init__() is not picklable in Windows. Read 16.6 multiprocessing - Programming guidelines - Windows .

Regarding your error with a top-level function - I suspect the way you defined it, it is generated differently each time and thus is really "not the same object" in child. I suggest passing a simple top-level function that would call run_testcase using reflection if you really need this level of sophistication. Update: this didn't help

Update:

I made this work by undecorating run_testcase,run_in_parallel and capture. @check decorator was replaced with def wrapper(function,*args) with the same functionality:

import traceback
def wrapper(function,*args):
    try:
        recorder=Recorder()
        PythonHelper().run_in_parallel([function,args],[recorder.capture])
        print("success")
    except Exception,e:
        print("failure: "+traceback.format_exc(10))

main:

from w_Decorators import wrapper

if __name__=="__main__":
    logger=Logger()
    wrapper(logger.run_testcase)

Just as I thought - decorated objects are not picklable.