When I run the setuptools .egg “as if it were a sh

Question

From reading this documentation, I've built a mental model of what the command sh setuptools-0.6c11-py2.7.eggactually does, but it's very incomplete and I'm still mystified by a few aspects.

My mental model goes something like this:

1. When this command is issued, the egg (which I'd considered as a sort of zip file that handles dependencies cleverly) somehow finds the right version of python on my system, and using that python, "installs itself" to an appropriate location.
2. In practice this means that a 'Unix Executable File' called easy_install is created in a directory that's hopefully on my path. This is why I can just type easy_install somepackage in the terminal afterwards.

My questions are therefore:

1. How is an 'egg' able to "install itself" in this way? Why does this work for this egg in particular, when other eggs require easy_install to work?
2. The 'Unix Executable File' thus created is 4kb. What actually is it? Is it full of calls to other things? Where are they?

This isn't a 'how to get it working' question - I'm not having any problems along that axis, but I'd like to fully grok what's going on here.

Answer 1

Egg files are simply zip-compressed directories containing Python packages, modules and a little metadata, with a .egg extension.

The zip format is flexible; it will ignore anything at the start of the file that isn't part of the zipfile. The zipfile is detected by finding a series of characters (PK and two more bytes indicating the type) and reading from there.

This means you can put something in front of the zip. The setuptools eggs are special in that they use this trick to insert some shell script before the zip data:

$ head -n8 setuptools-0.6c11-py2.7.egg 
#!/bin/sh
if [ `basename $0` = "setuptools-0.6c11-py2.7.egg" ]
then exec python2.7 -c "import sys, os; sys.path.insert(0, os.path.abspath('$0')); from setuptools.command.easy_install import bootstrap; sys.exit(bootstrap())" "$@"
else
  echo $0 is not the correct name for this egg file.
  echo Please rename it back to setuptools-0.6c11-py2.7.egg and try again.
  exec false
fi

Past those first 8 lines, it's a genuine zip:

$ tail -n+9 setuptools-0.6c11-py2.7.egg | file -
/dev/stdin: Zip archive data, at least v2.0 to extract

The script contained in those first 8 lines is what is being executed when you run sh setuptools-0.6c11-py2.7.egg.

As you can see, the script is specific to the egg version; this in the version for python 2.7, and it simply uses the python interpreter to add this egg to the python search path. It then imports a function from the contained python module and runs it.

Here are the contents of the egg itself, which is as I mentioned just a zip archive:

$ zipinfo -l setuptools-0.6c11-py2.7.egg 
Archive:  setuptools-0.6c11-py2.7.egg   332005 bytes   78 files
-rw-rw-r--  2.0 unx     1713 b-      995 defN  7-Jul-10 20:26 site.pyc
-rw-rw-r--  2.0 unx    90113 b-    31461 defN  7-Jul-10 20:26 pkg_resources.pyc
-rw-rw-r--  2.0 unx    85435 b-    23537 defN 19-Oct-09 13:35 pkg_resources.py
-rw-rw-r--  2.0 unx     2362 b-      875 defN 20-Sep-06 17:05 site.py
-rw-rw-r--  2.0 unx      309 b-      224 defN  7-Jul-10 20:26 easy_install.pyc
-rw-rw-r--  2.0 unx      126 b-      105 defN 20-Sep-06 17:05 easy_install.py
-rw-rw-r--  2.0 unx       43 b-       43 defN  7-Jul-10 20:26 EGG-INFO/top_level.txt
-rw-rw-r--  2.0 unx     1591 b-      458 defN  7-Jul-10 20:26 EGG-INFO/SOURCES.txt
-rw-rw-r--  2.0 unx        1 b-        3 defN 20-Oct-09 10:07 EGG-INFO/zip-safe
-rw-rw-r--  2.0 unx     9278 b-     3194 defN  7-Jul-10 20:26 EGG-INFO/PKG-INFO
-rwxrwxr-x  2.0 unx     2504 b-      623 defN  7-Jul-10 20:26 EGG-INFO/entry_points.txt
-rw-rw-r--  2.0 unx        1 b-        3 defN  7-Jul-10 20:26 EGG-INFO/dependency_links.txt
-rw-rw-r--  2.0 unx     1567 b-      791 defN  7-Jul-10 20:26 setuptools/extension.pyc
-rw-rw-r--  2.0 unx     1089 b-      424 defN 20-Sep-06 17:05 setuptools/extension.py
-rw-rw-r--  2.0 unx    10796 b-     4050 defN  7-Jul-10 20:26 setuptools/sandbox.pyc
-rw-rw-r--  2.0 unx     8227 b-     2309 defN  6-Jul-10 20:09 setuptools/sandbox.py
-rw-rw-r--  2.0 unx     5677 b-     2499 defN  7-Jul-10 20:26 setuptools/archive_util.pyc
-rw-rw-r--  2.0 unx    26800 b-    11228 defN  7-Jul-10 20:26 setuptools/package_index.pyc
-rw-rw-r--  2.0 unx     6209 b-     2229 defN 19-Oct-09 13:35 setuptools/depends.py
-rw-rw-r--  2.0 unx     6677 b-     3096 defN  7-Jul-10 20:26 setuptools/depends.pyc
-rw-rw-r--  2.0 unx     2816 b-     1159 defN  6-Jul-10 20:09 setuptools/__init__.py
-rw-rw-r--  2.0 unx     3639 b-     1837 defN  7-Jul-10 20:26 setuptools/__init__.pyc
-rw-rw-r--  2.0 unx     5924 b-     1777 defN 19-Oct-09 13:35 setuptools/archive_util.py
-rw-rw-r--  2.0 unx    29972 b-     8156 defN 19-Oct-09 13:35 setuptools/dist.py
-rwxrwxr-x  2.0 unx     7168 b-     3249 defN 19-Oct-09 17:18 setuptools/cli.exe
-rw-rw-r--  2.0 unx    28275 b-     8698 defN  6-Jul-10 20:09 setuptools/package_index.py
-rw-rw-r--  2.0 unx    29786 b-    10953 defN  7-Jul-10 20:26 setuptools/dist.pyc
-rwxrwxr-x  2.0 unx     7168 b-     3244 defN 19-Oct-09 17:18 setuptools/gui.exe
-rw-rw-r--  2.0 unx    22219 b-     7042 defN  7-Jul-10 20:26 setuptools/tests/test_resources.pyc
-rw-rw-r--  2.0 unx    19388 b-     4723 defN 24-Sep-08 13:10 setuptools/tests/test_resources.py
-rw-rw-r--  2.0 unx    12345 b-     2765 defN 24-Sep-08 13:10 setuptools/tests/__init__.py
-rw-rw-r--  2.0 unx    13811 b-     4523 defN  7-Jul-10 20:26 setuptools/tests/__init__.pyc
-rw-rw-r--  2.0 unx     1499 b-      708 defN  7-Jul-10 20:26 setuptools/tests/test_packageindex.pyc
-rw-rw-r--  2.0 unx    81351 b-    27171 defN  7-Jul-10 20:26 setuptools/tests/doctest.pyc
-rw-rw-r--  2.0 unx      759 b-      346 defN 24-Sep-08 13:10 setuptools/tests/test_packageindex.py
-rw-rw-r--  2.0 unx    99714 b-    25663 defN 20-Sep-06 17:05 setuptools/tests/doctest.py
-rw-rw-r--  2.0 unx     2866 b-     1332 defN  7-Jul-10 20:26 setuptools/command/rotate.pyc
-rw-rw-r--  2.0 unx    11520 b-     3127 defN 19-Oct-09 13:35 setuptools/command/build_ext.py
-rw-rw-r--  2.0 unx     6649 b-     2208 defN 24-Sep-08 13:10 setuptools/command/upload.py
-rw-rw-r--  2.0 unx     8162 b-     3538 defN  7-Jul-10 20:26 setuptools/command/sdist.pyc
-rw-rw-r--  2.0 unx     5965 b-     2421 defN  7-Jul-10 20:26 setuptools/command/setopt.pyc
-rw-rw-r--  2.0 unx     2283 b-      695 defN 19-Oct-09 17:50 setuptools/command/bdist_wininst.py
-rw-rw-r--  2.0 unx     7535 b-     3208 defN  7-Jul-10 20:26 setuptools/command/build_py.pyc
-rw-rw-r--  2.0 unx     3690 b-     1528 defN  7-Jul-10 20:26 setuptools/command/install.pyc
-rw-rw-r--  2.0 unx    14205 b-     4465 defN 19-Oct-09 13:35 setuptools/command/egg_info.py
-rw-rw-r--  2.0 unx      626 b-      311 defN 28-Dec-06 19:52 setuptools/command/__init__.py
-rw-rw-r--  2.0 unx      839 b-      494 defN  7-Jul-10 20:26 setuptools/command/__init__.pyc
-rw-rw-r--  2.0 unx     5053 b-     1519 defN 20-Sep-06 17:05 setuptools/command/setopt.py
-rw-rw-r--  2.0 unx      674 b-      329 defN  7-Jul-10 20:26 setuptools/command/register.pyc
-rw-rw-r--  2.0 unx     3724 b-     1292 defN  4-Sep-07 00:11 setuptools/command/install_egg_info.py
-rw-rw-r--  2.0 unx    18005 b-     5444 defN 19-Oct-09 13:35 setuptools/command/bdist_egg.py
-rw-rw-r--  2.0 unx     3984 b-     1385 defN 15-Feb-08 12:29 setuptools/command/install.py
-rw-rw-r--  2.0 unx     2356 b-     1002 defN  7-Jul-10 20:26 setuptools/command/bdist_wininst.pyc
-rw-rw-r--  2.0 unx     2025 b-      774 defN 22-May-07 17:55 setuptools/command/bdist_rpm.py
-rw-rw-r--  2.0 unx     2486 b-      871 defN 20-Sep-06 17:05 setuptools/command/install_lib.py
-rw-rw-r--  2.0 unx      740 b-      357 defN 20-Sep-06 17:05 setuptools/command/saveopts.py
-rw-rw-r--  2.0 unx    56980 b-    23198 defN  7-Jul-10 20:26 setuptools/command/easy_install.pyc
-rw-rw-r--  2.0 unx     3172 b-     1438 defN  7-Jul-10 20:26 setuptools/command/install_lib.pyc
-rw-rw-r--  2.0 unx     2257 b-     1013 defN  7-Jul-10 20:26 setuptools/command/bdist_rpm.pyc
-rw-rw-r--  2.0 unx     5310 b-     1732 defN 15-Feb-08 12:29 setuptools/command/develop.py
-rw-rw-r--  2.0 unx     5091 b-     2222 defN  7-Jul-10 20:26 setuptools/command/test.pyc
-rw-rw-r--  2.0 unx    63580 b-    17507 defN 19-Oct-09 13:35 setuptools/command/easy_install.py
-rw-rw-r--  2.0 unx    16467 b-     6544 defN  7-Jul-10 20:26 setuptools/command/egg_info.pyc
-rw-rw-r--  2.0 unx     4577 b-     1994 defN  7-Jul-10 20:26 setuptools/command/install_egg_info.pyc
-rw-rw-r--  2.0 unx     6275 b-     3108 defN  7-Jul-10 20:26 setuptools/command/upload.pyc
-rw-rw-r--  2.0 unx     7246 b-     2237 defN 20-Sep-06 17:05 setuptools/command/build_py.py
-rw-rw-r--  2.0 unx    10073 b-     4314 defN  7-Jul-10 20:26 setuptools/command/build_ext.pyc
-rw-rw-r--  2.0 unx     3185 b-     1463 defN  7-Jul-10 20:26 setuptools/command/alias.pyc
-rw-rw-r--  2.0 unx     1921 b-      704 defN 15-Feb-08 12:29 setuptools/command/install_scripts.py
-rw-rw-r--  2.0 unx     7327 b-     2380 defN 19-Oct-09 15:46 setuptools/command/sdist.py
-rw-rw-r--  2.0 unx     1249 b-      647 defN  7-Jul-10 20:26 setuptools/command/saveopts.pyc
-rw-rw-r--  2.0 unx     2021 b-      750 defN 20-Sep-06 17:05 setuptools/command/rotate.py
-rw-rw-r--  2.0 unx     2477 b-      848 defN 19-Oct-09 13:35 setuptools/command/alias.py
-rw-rw-r--  2.0 unx    17695 b-     7800 defN  7-Jul-10 20:26 setuptools/command/bdist_egg.pyc
-rw-rw-r--  2.0 unx      277 b-      158 defN 20-Sep-06 17:05 setuptools/command/register.py
-rw-rw-r--  2.0 unx     4442 b-     1424 defN 15-Feb-08 12:29 setuptools/command/test.py
-rw-rw-r--  2.0 unx     2445 b-     1160 defN  7-Jul-10 20:26 setuptools/command/install_scripts.pyc
-rw-rw-r--  2.0 unx     5175 b-     2317 defN  7-Jul-10 20:26 setuptools/command/develop.pyc
78 files, 958981 bytes uncompressed, 321419 bytes compressed:  66.5%