Copy flask request/app context to another process

问题:

tl;dr

How can I serialise a Flask app or request context, or a subset of that context (i.e. whatever can be successfully serialised) so that I can access that context from another process, rather than a thread?

Long version

I have some functions that require access to the Flask request context, or the App context, that I want to run in the background.

Flask has a built-in @copy_current_request_context decorator to wrap a function in a copy of the request context, so you can run it in a different thread:

from threading import Thread
from flask import Flask, request, copy_current_request_context

app = Flask(__name__)

@app.route('/')
def index():
    request.foo = 'bar'
    @copy_current_request_context
    def baz():
        print(request.foo)
    thr = Thread(target=baz)
    thr.start()
    return 'ok'

Whilst Flask doesn't provide a built-in decorator to copy the app context, it provides the machinery to do it - a solution is described on Access flask.g inside greenlet

What I've tried

First of all, I was hoping to solve with Pickle. This is what is used by concurrent.futures.ProcessPoolExecutor. Unfortunately, Pickle fails due to the presence of thread lock objects inside the app context:

>>> with app.test_request_context('/'):
...     pickle.dumps(appctx)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: can't pickle _thread.lock objects

It also can't copy decorated/wrapped functions:

>>> with app.test_request_context('/'):
...     bar = copy_current_request_context(pow)
...     pickle.dumps(bar)
...
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
_pickle.PicklingError: Can't pickle <function pow at 0x110bee8c8>: it's not the same object as builtins.pow

Next I tried dill. It passes the above two tests, but it can't serialise a lot of other things that might end up in a context. App contexts especially are prone to having extensions link themselves in: SQLAlchemy is a great example. Here's what happens when you're using SQLAlchemy and you try to serialise your app context:

>>> from flask_sqlalchemy import SQLAlchemy
>>> db = SQLAlchemy(app)
>>> with app.test_request_context('/'):
...     from flask.globals import _app_ctx_stack
...     appctx = _app_ctx_stack.top
...     dill.dumps(appctx)
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
    pik.dump(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1386, in save_function
    obj.__dict__), obj=obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1129, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global
    (obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.SignallingSession'>: it's not found as sqlalchemy.orm.session.SignallingSession

So: is there any way around this? One thought I had was that forking a new process "just in time" - i.e. immediately before I want to run my background task - may mean I don't even need to copy a context at all, it should already have it as long as the fork happens after the context is set up. I don't know how to make this work with a ProcessPoolExecutor however, where processes are a long lived pool. The only idea I have there is to somehow force worker processes to shut down at the end of each task but I'm pretty sure that this will just result in a broken pool.