python shell: pickle entire state

Question

When using either "ipython" or "code.interact(local=locals())", I'd like to have a way to save the entire program address space into a pickle file, and similarly a way to load such a file and then start executing in that context.

This should be totally possible for simple scripts because of virtual memory.

All defined names (and the non-orphaned objects they point to), from locals up through globals and global functions, would be pickled. When unpickling, those names would be assigned again, at their original scope.

Assume that the program doesn't use the network, so state discontinuity there is avoided.

Assume that the program is fault-tolerant with regards to the perceived discontitunity of the system clock, so no problem there either.

The only challenge seems to be what to do with file descriptors. Ideally, if this doesn't exist already, there should be a simple "file descriptor pickle" function that gets the mode bits it is open with, and the postion of the file cursor, and a checksum of the file contents (erroring if the checksums don't match when trying to unpickle).

Are there a few lines that will accomplish this "pickling the entire session"?

Answer 1

To do this, I'd use dill, which can serialize almost anything in python.

>>> import dill
>>> 
>>> def foo(a):
...   def bar(x):
...     return a*x
...   return bar
... 
>>> class baz(object):
...   def __call__(self, a,x):
...     return foo(a)(x)
... 
>>> b = baz()
>>> b(3,2)
6
>>> c = baz.__call__
>>> c(b,3,2)
6
>>> g = dill.loads(dill.dumps(globals()))
>>> g
{'dill': <module 'dill' from '/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <module '__builtin__' (built-in)>, 'baz': <class '__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x4d39d30>, '__doc__': None}

Dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

You also wanted to pickle the whole interpreter session... dill can do that too.

>>> # continuing from above
>>> dill.dump_session('foobar.pkl')
>>>
>>> ^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foobar.pkl')
>>> c(b,3,2)
6

Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.