Python *.py, *.pyo, *.pyc: Which can be eliminated

Question

To squeeze into the limited amount of filesystem storage available in an embedded system I'm currently playing with, I would like to eliminate any files that could reasonably be removed without significantly impacting functionality or performance. The *.py, *.pyo, and *.pyc files in the Python library account for a sizable amount of space, I'm wondering which of these options would be most reasonable for a Python 2.6 installation in a small embedded system:

1. Keep *.py, eliminate *.pyc and *.pyo (Maintain ability to debug, performance suffers?)
2. Keep *.py and *.pyc, eliminate *.pyo (Does optimization really buy anything?)
3. Keep *.pyc, eliminate *.pyo and *.py (Will this work?)
4. Keep *.py, *.pyc, and *.pyo (All are needed?)

Answer 1

Here's how I minimize disk requirements for mainline Python 2.7 at the day job:

1) Remove packages from the standard library which you won't need. The following is a conservative list:

bsddb/test ctypes/test distutils/tests email/test idlelib lib-tk
lib2to3 pydoc.py tabnanny.py test unittest

Note that some Python code may have surprising dependencies; e.g. setuptools needs unittest to run.

2) Pre-compile all Python code, using -OO to strip asserts and docstrings.

find -name '*.py' | python -OO -m py_compile -

Note that Python by default does not look at .pyo files; you have to explicitly ask for optimization at runtime as well, using an option or an environment variable. Run scripts in one of the following ways:

python -OO -m mylib.myscript
PYTHONOPTIMIZE=2 python -m mylib.myscript

3) Remove .py source code files (unless you need to run them as scripts) and .pyc unoptimized files.

find '(' -name '*.py' -or -name '*.pyc' ')' -and -not -executable -execdir rm '{}' ';'

4) Compress the Python library files. Python can load modules from a zip file. The paths in the zip-file must match the package hierarchy; thus you should merge site-packages and .egg directories into the main library directory before zipping. (Or you can add multiple zip files to the Python path.)

On Linux, Python's default path includes /usr/lib/python27.zip already, so just drop the zip file there and you're ready to go.

Leave os.pyo as an ordinary (non-zipped) file, since Python looks for this as a sanity check. If you move it to the zip file, you'll get a warning on every Python invocation (though everything will still work). Or you can just leave an empty os.py file there, and put the real one in the zip file.

Final notes:

• In this manner, Python fits in 7 MB of disk space. There's a lot more that can be done to reduce size, but 7 MB was small enough for my purposes. :)
• Python bytecode is not compatible across versions, but who cares when it's you who do the compilation and you who controls the Python version?
• .pyo files in a zip file should be a performance win in all cases, unless the disk is extremely fast and the processor/RAM is extremely slow. Either way, Python executes from memory, not the on-disk format, so it only affects performance on load. Although the stripping of docstrings can save quite a bit of memory.
• Do note that .pyo files do not contain assert statements.
• .pyo files preserve function names and line numbers, so debugability is not decreased: You still get nice tracebacks, you just have to manually go look up the line number in the source, which you'd have to do anyway.
• If you want to "hack" a file at runtime, just put it in the current working directory. It take precedence over the library zip file.

Answer 2

I would recommend keeping only .py files. The difference in startup time isn't that great, and having the source around is a plus, as it will run under different python versions without any issues.

As of python 2.6, setting sys.dont_write_bytecode to True will suppress compilation of .pyc and .pyo files altogether, so you may want to use that option if you have 2.6 available.

Answer 3

http://www.network-theory.co.uk/docs/pytut/CompiledPythonfiles.html

When the Python interpreter is invoked with the -O flag, optimized code is generated and stored in ‘.pyo’ files. The optimizer currently doesn't help much; it only removes assert statements.

Passing two -O flags to the Python interpreter (-OO) will cause the bytecode compiler to perform optimizations that could in some rare cases result in malfunctioning programs. Currently only doc strings are removed from the bytecode, resulting in more compact ‘.pyo’ files.

My suggestion to you?

Use -OO to compile only .pyo files if you don't need assert statements and __doc__ strings.

Otherwise, go with .pyc only.

Edit

I noticed that you only mentioned the Python library. Much of the python library can be removed if you only need part of the functionality.

I also suggest that you take a look at tinypy which is large subset of Python in about 64kb.

Answer 4

What it ultimately boils down to is that you really only need one of the three options, but your best bet is to go with .pys and either .pyos or .pycs.

Here's how I see each of your options:

1. If you put the .pys in a zip file, you won't see pycs or pyos built. It should also be pointed out that the performance difference is only in startup time, and even then isn't too great in my experience (your milage may vary though). Also note that there is a way to prevent the interpreter from outputting .pycs as Algorias points out.
2. I think that this is an ideal option (either that or .pys and .pyos) because you get the best mix of performance, debuggability and reliability. You don't necessarily need a source file and compiled file though.
3. If you're really strapped for space and need performance, this will work. I'd advise you to keep the .pys if at all possible though. Compiled binaries (.pycs or .pyos) don't always transfer to different versions of python.
4. It's doubtful that you'll need all three unless you plan on running in optimized mode sometimes and non-optimized mode sometimes.

In terms of space it's been my (very anecdotal) experience that .py files compress the best compared to .pycs and .pyos if you put them in a zipfile. If you plan on compressing the files, .pyos don't tend to gain a lot in terms of sheer space because docstrings tend to compress fairly well and asserts just don't take up that much space.

Answer 5

Number 3 should and will work. You do not need the .pyo or .py files in order to use the compiled python code.