How can I correctly install multiple non-package D

Question

I am developing Python applications in Ubuntu. I want to setup a Distribute/virtualenv/pip ecosystem to manage my Python packages independently of any system Python packages (which I manage in Synaptic, or rather I let the system manage them for me).

I could just install the python-setuptools, python-virtualenv and python-pip system packages and be on my merry way, but I also want to be able to get latest/specific versions of Distribute, virtualenv and pip. There are no PPAs for these, so I'll have to install them manually.

A final complication, is that I want to be able to do this for multiple versions of Python. That is, set up one ecosystem for python2.6, another for python, another for python3, or on a 64-bit system another for chrooted 32-bit Python.

I'm guessing that the process would be something like:

• Using Python X install my own copy of Distribute to a location in my home folder
• Using indie Distribute, easy_install pip
• Using indie pip, install virtualenv
• Using indie virtualenv, create virtual environment
• Activate virtual environment, install packages
• Repeat for Python Y, Z and Q

What installation/configuration options am I looking for?

Answer 1

Based on Walker Hale IV's answer to a similar (but distinct! ;) ) question, there are two keys to doing this:

• you don't need to install distribute and pip because these are automatically included in a new virtual environment (and you presumably only want the versions that have been tested with virtualenv)
• you can use the virtualenv source code to create a new virtual environment, rather than using the system-installed version of virtualenv

So the workflow is:

• install Python version X on your system
• download virtualenv source code version Q (probably the latest)
• create new virtual environment with Python X and virtualenv Q
• your new VE is now running Python X and the latest stable versions of pip and distribute
• pip install any other packages
• easy_install any other packages that you can't pip install

Notes:

• In your new virtual environment, you could install new (or old) versions of distribute, pip or virtualenv. (I think)
• I don't use WH4's technique of create a bootstrap virtual environment. Instead, I create the new virtual environment from the virtualenv source each time.
• This technique should be usable on any operating system.
• If I were explaining this to someone new to the whole Distribute/pip/virtualenv ecosystem concept, I would explain it in a virtualenv-centric way.

I've written a bash script that does the basics in Ubuntu:

#! /bin/bash
# Script to create a python virtual environment
# independently of the system version of virtualenv
#
# Ideally this would be a cross-platform Python
# script with more validation and fewer TODOs,
# but you know how it is.

# = PARAMETERS =
# $1 is the python executable to use
# examples: python, python2.6, /path/to/python
# $2 is the new environment folder
# example: /path/to/env_folder/name
## TODO: should be just a name
## but I can't concatenate strings in bash
# $3 is a pip requirements file
# example: /path/to/req_folder/name.txt
# you must uncomment the relevant code below to use $3
## TODO: should be just a name
## but I can't concatenate strings in bash

# other parameters are hard-coded below
# and you must change them before first use

# = EXAMPLES OF USE =
# . env_create python2.5 /home/env/legacy
## creates environment "legacy" using python 2.5
# . env_create python /home/env/default
## creates environment "default" using whatever version of python is installed
# . env_create python3.2 /home/env/bleeding /home/req/testing.txt
## creates environment "bleeding" using python 3.2 and installs packages from testing.txt using pip

# = SET UP VARIABLES =
# Required version of virtualenv package
VERSION=1.6.4
# Folder to store your virtual environments
VE_FOLDER='/media/work/environments'
## TODO: not used because I can't concatenate strings in bash
# Folder to store bootstrap (source) versions of virtualenv
BOOTSTRAP_FOLDER='/media/work/environments/bootstrap'
## TODO: not used because I can't concatenate strings in bash
# Folder to store pip requirements files
REQUIREMENTS_FOLDER='/media/work/environments/requirements'
## TODO: not used because I can't concatenate strings in bash
# Base URL for downloading virtualenv source
URL_BASE=http://pypi.python.org/packages/source/v/virtualenv
# Universal environment options
ENV_OPTS='--no-site-packages --distribute'
# $1 is the python interpreter
PYTHON=$1
# $2 is the target folder of the new virtual environment
VE_TARGET=$2
# $3 is the pip requirements file
REQ_TARGET=$3

## = DOWNLOAD VIRTUALENV SOURCE =
## I work offline so I already have this downloaded
## and I leave this bit commented out
# cd $BOOTSTRAP_DIR
# curl -O $URL_BASE/virtualenv-$VERSION.tar.gz
## or use wget

# = CREATE NEW ENV USING VIRTUALENV SOURCE =
cd $BOOTSTRAP_FOLDER
tar xzf virtualenv-$VERSION.tar.gz
# Create the environment
$PYTHON virtualenv-$VERSION/virtualenv.py $ENV_OPTS $VE_TARGET
# Don't need extracted version anymore
rm -rf virtualenv-$VERSION
# Activate new environment
cd $VE_TARGET
. bin/activate

# = INSTALL A PIP REQUIREMENTS FILE =
## uncomment this if you want to automatically install a file
# pip install -r $REQ_TARGET

# = REPORT ON THE NEW ENVIRONMENT =
python --version
pip freeze
# deactivate
## uncomment this if you don't want to start in your environment immediately

Output looks something like this (with downloading switched off and deactivation switched on):

user@computer:/home/user$ . env_create python3 /media/work/environments/test
New python executable in /media/work/environments/test/bin/python3
Also creating executable in /media/work/environments/test/bin/python
Installing distribute...............done.
Installing pip...............done.
Python 3.2
distribute==0.6.19
wsgiref==0.1.2
user@computer:/media/work/environments/test$ 

Answer 2

Elaborating on JF Sebastian and nealmcb's contributions, these days I do indeed use my system packaged version of virtualenvwrapper (available on Ubuntu 12.04 and later).

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

The key features I use (in answer to this question) are:

• mkvirtualenv --python=PYTHON_EXE creates a virtualenv using a specific Python executable (doesn't have to be a system packaged version)
• tie in to pip's virtualenv support
• allvirtualenv pip install -U pip upgrades pip in all virtualenvs

The environment variables JFS mentioned are indeed useful to fiddle with: PIP_DOWNLOAD_CACHE, VIRTUALENV_USE_DISTRIBUTE, WORKON_HOME, VIRTUALENVWRAPPER_PYTHON.

The only reason to update virtualenv itself is to get the latest version of setuptools (previously known as Distribute, previously known as setuptools). I haven't had a need to do this yet, but I suspect it would be easiest to start with a fresh virtualenv and upgrade Distribute/setuptools first, then upgrade pip, then install other libraries.

If a new version of virtualenv is strictly necessary, a modification of the bootstrap script should do.

Answer 3

As noted by @j.f.sebastian, virtualenvwrapper does much or all of what you're asking for.

http://virtualenvwrapper.readthedocs.org/

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.