ValueError: insecure string pickle

Question

When I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.

How could I solve this problem?

Answer 1

If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695

richiecannizzo commented on Aug 28

brew install libav
Should fix it instantly on mac or

sudo apt-get install libav
#on linux

Answer 2

What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.

Answer 3

Check this thread. Peter Otten says:

A corrupted pickle. The error is raised if a string in the dump does not both start and end with " or '.

and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).

Answer 4

Same problem with a file that was made with python on windows, and reloaded with python on linux. Solution : dos2unix on the file before reading in linux : works as a charm !

Answer 5

I got the Python ValueError: insecure string pickle message in a different way.

For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.

Initially I was wrapping up a file like this

with open(path_to_binary_file) as data_file:
    contents = data_file.read() 
filename = os.path.split(path)[1]

url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()

server_response = urllib2.urlopen(url, base64_message)

But on the server the hash kept coming out differently for some binary files

decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()

And unpickling gave insecure string pickle message

cPickle.loads(decoded_message)

BUT SUCCESS

What worked for me was to use urlsafe_b64encode()

base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))

And decode with

base64_decoded_message = base64.urlsafe_b64decode(base64_message)

References

http://docs.python.org/2/library/base64.html

https://tools.ietf.org/html/rfc3548.html#section-3

Answer 6

I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)

    # Push file to another machine
    _send_file(temp_file.name)

However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.

Two potential fixes to this: Keep the file open and force a flush:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)
    temp_file.flush()

    # Push file to another machine
    _send_file(temp_file.name)

Or make sure the file is closed before doing anything with it:

file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
    file_name = temp_file.name
    pickle.dump(foo, temp_file)

# Push file to another machine
_send_file(file_name)