What's the exact usage of __reduce__ in Pickle

Question

I know that in order to be picklable, a class has to overwrite __reduce__ method, and it has to return string or tuple.

How does this function work? What the exact usage of __reduce__? When will it been used?

Answer 1

I'll see if I can take a stab at explaining this one.

Whenever you try to pickle an object, there will be some properties that may not serialize well. For instance, an open file handle In this cases, pickle won't know how to handle the object and will throw an error.

You can tell the pickle module how to handle these types of objects natively within a class directly. Lets build the example of an object which has a single property; an open file handle:

import pickle

class test(object):
    def __init__(self, file_path = 'test1234567890.txt'):
        self.some_file_i_have_opened = open(file_path, 'wb')  # An open file in write mode.

my_test = test()
# Now, watch what happens when we try to pickle this object:
pickle.dumps(my_test)

It should fail, and give a traceback:

Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  --- snip snip a lot of lines ---
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects

However, had we defined a __reduce__ method in our test class, pickle would have known how to serialize this object:

import pickle

class test(object):
    def __init__(self, file_path = 'test1234567890.txt'):
        self._file_name_we_opened = file_path  # Used later in __reduce__
        self.some_file_i_have_opened = open(self._file_name_we_opened, 'wb')  # An open file in write mode.
    def __reduce__(self):
        return (self.__class__, (self._file_name_we_opened, ))  # we return a tuple of class_name to call, and optional parameters to pass when re-creating

my_test = test()
saved_object = pickle.dumps(my_test)
print repr(saved_object)  # Just print the representation of the string of the object, because it contains newlines.

This should give you something like: "c__main__\ntest\np0\n(S'test1234567890.txt'\np1\ntp2\nRp3\n.", which can be used to recreate the object with open file handles:

print vars(pickle.loads(saved_object))

Usually, the big confusion is with what type of object __reduce__ should return. while you can read a little more about what type of object reduce should return in the docs pages: http://aakashlabs.org/docs/apl/pyhelp/pydocs/library/pickle.html#pickling-and-unpickling-extension-types, but in general, it needs a tuple of at least 2 things:

1. A blank object class to call. In this case, self.__class__
2. A tuple of arguments to pass to the class constructor. In this case, it's a single string, which is the path to the file to open.

There are other optional items, but you should read all about them in the docs.

Hope that helps!