I have pickled files using protocol 3 in python3,

Question

mydata = pickle.load(myfile, "rb")
ValueError: unsupported pickle protocol: 3

This is related to: ValueError: unsupported pickle protocol: 3, python2 pickle can not load the file dumped by python 3 pickle?

So clearly, with foresight, you must set protocol=2 when you dump your pickle file in python3 if you want to be able to unpickle it with python2.

However what if you unfortunately are stuck with files that were pickled in python 3 using protocol 3 and now you must read them with python2? Is there any workaround?

There is a related question, but it seems to be a different core issue: unpickle OrderedDict from python3 in python2

Answer 1

The simplest and easiest approach will be to write a Python3 script that unpickles everything using protocol 3 and repickles it again using protocol 2. (Consider switching to camel at the same time.)

In Python 3:

pickle.dump(pickle.load(sys.stdin), sys.stdout, 2)

Then in Python 2:

pickle.load(...) # This will work now in Python 2.

Answer 2

If you have no control over the way these pickled files were created, and you must load them in Python 2.7, then unfortunately there is no easy workaround.

Perhaps the person who created the files wasn’t aware of the fact that the default protocol for pickle in Python 3 is “3”, but that this is backward-incompatible.

From the documentation:

Currently the default protocol is 3; a backward-incompatible protocol designed for Python 3.0.

If they are a fixed set of files then a workaround may be to just write a script that iteratively loads the protocol 3 pickled files in Python 3.0 and then re-writes them using protocol=2 as one-shot fix. Then you will be able to read them in Python 2.7. Also make sure that the original code writing these files is modified if there are future files that are going to be created that your code will need to handle.

@Kay points out how simple this solution is in practice: I have pickled files using protocol 3 in python3, and now I need to unpickle them with python2, what can I do?

As easy as pickle.dump(pickle.load(sys.stdin), sys.stdout, 2)