ValueError: insecure string pickle

问题:

When I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.

How could I solve this problem?

回答1:

"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.

One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: insecure string pickle

Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))

Easy to forget. Didn't need people being told that they are idiots.

回答2:

Check this thread. Peter Otten says:

A corrupted pickle. The error is raised if a string in the dump does not both start and end with " or '.

and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).

回答3:

What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.

回答4:

I've get this error in Python 2.7 because of open mode 'rb':

    with open(path_to_file, 'rb') as pickle_file:
        obj = pickle.load(pickle_file)

So, for Python 2 'mode' should be 'r'

Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:

pickle.unpicklingerror: the string opcode argument must be quoted

回答5:

If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695

richiecannizzo commented on Aug 28

brew install libav
Should fix it instantly on mac or

sudo apt-get install libav
#on linux

回答6:

This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):

Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: insecure string pickle

This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.

回答7:

Same problem with a file that was made with python on windows, and reloaded with python on linux. Solution : dos2unix on the file before reading in linux : works as a charm !

回答8:

I got the Python ValueError: insecure string pickle message in a different way.

For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.

Initially I was wrapping up a file like this

with open(path_to_binary_file) as data_file:
    contents = data_file.read() 
filename = os.path.split(path)[1]

url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()

server_response = urllib2.urlopen(url, base64_message)

But on the server the hash kept coming out differently for some binary files

decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()

And unpickling gave insecure string pickle message

cPickle.loads(decoded_message)

BUT SUCCESS

What worked for me was to use urlsafe_b64encode()

base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))

And decode with

base64_decoded_message = base64.urlsafe_b64decode(base64_message)

References

http://docs.python.org/2/library/base64.html

https://tools.ietf.org/html/rfc3548.html#section-3

回答9:

This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:

Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.

The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.

回答10:

I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)

    # Push file to another machine
    _send_file(temp_file.name)

However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.

Two potential fixes to this: Keep the file open and force a flush:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)
    temp_file.flush()

    # Push file to another machine
    _send_file(temp_file.name)

Or make sure the file is closed before doing anything with it:

file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
    file_name = temp_file.name
    pickle.dump(foo, temp_file)

# Push file to another machine
_send_file(file_name)