From virtualenv, pip freeze > requirements.txt giv

Question

I'm following this tutorial: http://devcenter.heroku.com/articles/django

At some point I'm suposed to do:

pip freeze > requirements.txt

(Ofc. from virtualenv created instance of python)

And I get this:

(venv)przemoli@ubuntu:~/Programowanie/hellodjango$ cat requirements.txt 
BeautifulSoup==3.2.0
Brlapi==0.5.5
CherryPy==3.1.2
ClientForm==0.2.10
Django==1.3
GnuPGInterface==0.3.2
PAM==0.4.2
PIL==1.1.7
Routes==1.12.3
Twisted-Core==11.0.0
Twisted-Names==11.0.0
Twisted-Web==11.0.0
WebOb==1.0.8
adium-theme-ubuntu==0.3.1
apt-xapian-index==0.44
apturl==0.5.1ubuntu1
chardet==2.0.1
command-not-found==0.2.44
configglue==1.0
cssutils==0.9.8a1
defer==1.0.2
distribute==0.6.19
django-tagging==0.3.1
dnspython==1.9.4
duplicity==0.6.15
gnome-app-install==0.4.7-nmu1ubuntu2
httplib2==0.7.2
jockey==0.9.4
keyring==0.6.2
launchpadlib==1.9.8
lazr.restfulclient==0.11.2
lazr.uri==1.0.2
louis==2.3.0
lxml==2.3
mechanize==0.1.11
nvidia-common==0.0.0
oauth==1.0.1
onboard==0.96.1
oneconf==0.2.6.7
papyon==0.5.5
pexpect==2.3
piston-mini-client==0.6
protobuf==2.4.0a
psycopg2==2.4.4
pyOpenSSL==0.12
pycrypto==2.3
pycups==1.9.59
pycurl==7.19.0
pyinotify==0.9.1
pyparsing==1.5.2
pyserial==2.5
pysmbc==1.0.10
python-apt==0.8.0ubuntu9
python-dateutil==1.4.1
python-debian==0.1.20ubuntu2
python-virtkey==0.60.0
pyxdg==0.19
sessioninstaller==0.0.0
simplejson==2.1.6
system-service==0.1.6
ubuntu-sso-client==1.4.0
ubuntuone-couch==0.3.0
ubuntuone-installer==2.0.0
ubuntuone-storage-protocol==2.0.0
ufw==0.30.1-2ubuntu1
unattended-upgrades==0.1
usb-creator==0.2.23
virtualenv==1.6.4
wadllib==1.2.0
wsgiref==0.1.2
xdiagnose==1.1
xkit==0.0.0
zope.interface==3.6.1

When deploying on heroku it fails at Brlapi .....

I see lots of stuff from my main python installation which is on ubuntu. Which is BAD since Ubuntu use python for quite a few thing itself (ubuntu-one, usb-creator, etc..).

I do not need them on heroku! I need only Django, psycopg2, and their dependencies. I do not even know if its fault of pip, or virutalenv. (If you want to know my setup look at link above I copied it into terminal)

Answer 1

That is one thing that has bugged me too quite a bit. This happens when you create a virtualenv without the --no-site-packages flag.

There are a couple of things you can do:

1. Create virtualenv with the --no-site-packages flag.
2. When installing apps, dont run pip install <name> directly, instead, add the library to your requirements.txt first, and then install the requirements. This is slower but makes sure your requirements are updated.
3. Manually delete libraries you dont need. A rule of thumb i follow for this is to add whatever is there in my INSTALLED_APPS, and database adapters. Most other required libraries will get installed automatically because of dependencies. I know its silly, but this is what I usually end up doing.

-- Edit --

I've since written a couple of scripts to help manage this. The first runs pip freeze and adds the found library to a provided requirements file, the other, runs pip install, and then adds it to the requirements file.

function pipa() {
    # Adds package to requirements file.
    # Usage: pipa <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    package_string=`pip freeze | grep -i $package_name`
    current_requirements=`cat $requirements_file`
    echo "$current_requirements\n$package_string" | LANG=C sort | uniq > $requirements_file
}

function pipia() {
    # Installs package and adds to requirements file.
    # Usage: pipia <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    pip install $package_name
    pipa $package_name $requirements_file
}

Answer 2

If you care a lot about the cleanliness of your requirements.txt you should not only use the --no-site-packages option as already mentioned but also consider not to pipe the output of pip freeze directly to your requirements.txt. The reason for that is, that when doing a pip freeze not only packages specified by yourself show up, but also dependencies installed by these packages! It isn't necessary to keep them all in your requirements.txt as they will get installed automatically with the package that requires them... So if you add a new package to your virtualenv you probably should just add the line for this package to your requirements.txt...

See this example:

(demo)[~]$ pip freeze
distribute==0.6.19
wsgiref==0.1.2
(demo)[~]$ pip install django-blog-zinnia
Downloading/unpacking django-blog-zinnia
  Downloading django-blog-zinnia-0.9.tar.gz (523Kb): 523Kb downloaded
  Running setup.py egg_info for package django-blog-zinnia

    no previously-included directories found matching 'docs/api'
    no previously-included directories found matching 'docs/build'
    no previously-included directories found matching 'docs/coverage'
    no previously-included directories found matching 'zinnia/media/zinnia/css/.sass-cache'
Downloading/unpacking BeautifulSoup>=3.2.0 (from django-blog-zinnia)
  Downloading BeautifulSoup-3.2.1.tar.gz
  Running setup.py egg_info for package BeautifulSoup

  # truncated as it installs some more dependencies
Successfully installed django-blog-zinnia BeautifulSoup django-mptt django-tagging django-xmlrpc pyparsing
Cleaning up...
(demo)[~]$ pip freeze
BeautifulSoup==3.2.1
distribute==0.6.19
django-blog-zinnia==0.9
django-mptt==0.5.2
django-tagging==0.3.1
django-xmlrpc==0.1.3
pyparsing==1.5.6
wsgiref==0.1.2

(Though I should probably mentioned that in most cases it will not hurt that you have these dependencies there, just your file will grow and get harder to maintain.)

Answer 3

pipreqs solves the problem. It generates project-level requirement.txt file.

• Install pipreqs: pip install pipreqs
• Generate project-level requirement.txt file: pipreqs /path/to/your/project/ requirements file would be saved in /path/to/your/project/requirements.txt