How to use dill to serialize a class definition?

Question

In the answer to Python pickle: dealing with updated class definitions, the author of the dill package writes:

"Ok, I have added this feature to dill in the latest revision on github. Implemented with far less trickery than I thought... just serialize the class definition with the pickle, and voila."

Having installed dill and tinkered with it, it's not obvious to me how to actually use this functionality in dill. Could someone provide an explicit example? I would like to pickle the class instance and also serialize the class definition.

(I am new to python and I this functionality seems extremely important, as since when pickling an object it would be great to get as close to a guarantee as possible that you could look at the object (could be the result of a simulation) in the future after the class definition may have changed and you haven't kept track of all the changes in an easily accessible way.)

Answer 1

If this functionality were that important, it would be in the language core by now. :-) So, no, this is not critical to use Python in any advanced form - and if you have a project that relies on being able to re-instantiate objects based in old models - which is possible, you have to carefully think about it, and probably keep the old models around in explicit code, instead of having then serialized.

My advice is just "leave that apart" until you think you actually need it, and had compared it with other solutions, like a strong model migration policy.

That said, I've tried dill, and it works as advertised: it can serialize a class just like pickle can do with ordinary objects using the "dump" and "dumps" calls, and rebuild the class object with "load" and "loads".

What probably is getting you confused is that serializing an object (through pickle or dill alike) does not include neither its source code (i.e. the actual lines of textual Python code used to define the class), nor its name.

So, if a class is named "A", when it is serialized, if you need that name after "undilling" it, you have to reassign that name to it in the global name space. It's orignal name is preserved in it's __name__ attribute. (and for your purposes of multiple versions of the same model living together, that would lead to a lot of conflict).

Thus:

class A(object):
    ...

import dill

dill.dump(A, open("myfile", "w"))

del A
....
someclass = dill.load(open("myfile"))
print (someclass.__name__)
globals()[someclass.__name__] = someclass
# at this point you have the "A" class back in the global namespace

Answer 2

I think you are looking for one of the following functionalities…

Here I build a class, and an instance, and then change the class definition. The pickled class and instance still are unpicklable because dill pickles the source code for the class by default… and manages having several classes with the same name in the namespace (it does this simply by managing the pointer references to the class definitions).

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> 
>>> class Foo(object):
...   def bar(self, x):
...     return x+self.y       
...   y = 1
... 
>>> f = Foo()
>>> _Foo = dill.dumps(Foo)
>>> _f = dill.dumps(f)
>>> 
>>> class Foo(object):
...   def bar(self, x):
...     return x*self.z  
...   z = -1 
... 
>>> f_ = dill.loads(_f, ignore=True)
>>> f_.y
1
>>> f_.bar(1)
2
>>> Foo_ = dill.loads(_Foo)
>>> g = Foo_()
>>> g.bar(1)
2

Pickle would blow up on the above. If you don't want dill to serialize the class explicitly, and to do what pickle does, then you can ask dill to pickle by reference with dill.dumps(Foo, byref=True). Alternately, you can dynamically decide to ignore the newly-defined class by using ignore=False (the default).

Now, in the case below, we work with the new class definition, and extract the source from the object, then save it to a file. Additionally, we can dump the source to a file (here I use a temporary file) so it can be imported later.

>>> sFoo = dill.source.getsource(Foo)
>>> print sFoo
class Foo(object):
  def bar(self, x):
    return x*self.z
  z = -1

>>> open('myFoo.py', 'w').write(sFoo)    
>>>
>>> f = dill.temp.dump_source(Foo, dir='.')
>>> f.name
'/Users/mmckerns/dev/tmpM1dzYN.py'
>>> from tmpM1dzYN import Foo as _Foo_
>>> h = _Foo_()
>>> h.bar(2)
-2
>>> from myFoo import Foo as _SFoo_
>>> _SFoo_.z
>>> -1
>>> 

I hope that helps.