Embed pickle (or arbitrary) data in python script

Question

In Perl, the interpreter kind of stops when it encounters a line with

__END__

in it. This is often used to embed arbitrary data at the end of a perl script. In this way the perl script can fetch and store data that it stores 'in itself', which allows for quite nice opportunities.

In my case I have a pickled object that I want to store somewhere. While I can use a file.pickle file just fine, I was looking for a more compact approach (to distribute the script more easily).

Is there a mechanism that allows for embedding arbitrary data inside a python script somehow?

Answer 1

With pickle you can also work directly on strings.

s = pickle.dumps(obj)
pickle.loads(s)

If you combine that with """ (triple-quoted strings) you can easily store any pickled data in your file.

Answer 2

In Python, you can use """ (triple-quoted strings) to embed long runs of text data in your program.

In your case, however, don't waste time on this.

If you have an object you've pickled, you'd be much, much happier dumping that object as Python source and simply including the source.

The repr function, applied to most objects, will emit a Python source-code version of the object. If you implement __repr__ for all of your custom classes, you can trivially dump your structure as Python source.

If, on the other hand, your pickled structure started out as Python code, just leave it as Python code.

Answer 3

I made this code. You run something like python comp.py foofile.tar.gz, and it creates decomp.py, with foofile.tar.gz's contents embedded in it. I don't think this is really portable with windows because of the Popen though.

import base64
import sys
import subprocess

inf = open(sys.argv[1],"r+b").read()
outs = base64.b64encode(inf)

decomppy = '''#!/usr/bin/python
import base64

def decomp(data):
    fname = "%s"
    outf = open(fname,"w+b")
    outf.write(base64.b64decode(data))
    outf.close()

    # You can put the rest of your code here.
    #Like this, to unzip an archive
    #import subprocess
    #subprocess.Popen("tar xzf " + fname, shell=True)
    #subprocess.Popen("rm " + fname, shell=True)
''' %(sys.argv[1])

taildata = '''uudata = """%s"""
decomp(uudata)
''' %(outs)

outpy = open("decomp.py","w+b")
outpy.write(decomppy)
outpy.write(taildata)
outpy.close()
subprocess.Popen("chmod +x decomp.py",shell=True)

Answer 4

If the data is not particularly large (many K) I would just .encode('base64') it and include that in a triple-quoted string, with .decode('base64') to get back the binary data, and a pickle.loads() call around it.