Stop pip from failing on single package when insta

Question

I am installing packages from requirements.txt

pip install -r requirements.txt

The requirements.txt file reads:

Pillow
lxml
cssselect
jieba
beautifulsoup
nltk

lxml is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml fails pip still runs through and downloads the rest of the packages.

From what I understand the pip install -r requirements.txt command will fail if any of the packages listed in the requirements.txt fail to install.

Is there any argument I can pass when running pip install -r requirements.txt to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?

Answer 1

Running each line with pip install may be a workaround.

cat requirements.txt | xargs -n 1 pip install

Note: -a parameter is not available under MacOS, so old cat is more portable.

Answer 2

For windows:

# This code install line by line a list of pip package

import sys
import pip # from pip._internal import main as pip_main (pip>=18)

def install(package):
    pip.main(['install', package]) # pip_main(['install', package]) (pip>=18)

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

edit: Add info for pip 18 and later thx Jaeyoon Jeong

Answer 3

Thanks, Etienne Prothon for windows cases.

But, after upgrading to pip 18, pip package don't expose main to public. So you may need to change code like this.

 # This code install line by line a list of pip package 
 import sys
 from pip._internal import main as pip_main

 def install(package):
    pip_main(['install', package])

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

Answer 4

This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.

cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install

Hat tip to this answer for the sed magic.

Answer 5

Do you have requirements for lxml using? Here they are for install:

sudo apt-get install libxml2-dev libxslt-dev python-dev

If you use Windows or Mac, you can check that too. Alternatively, setting STATIC_DEPS=true will download and build both libraries automatically.(c)
http://lxml.de/installation.html

Answer 6

The xargs solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.

As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends section of a debian package.

I use the following shell script (requires sed) to install optional dependencies:

#!/bin/sh

while read dependency; do
    dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
    # Skip comments
    if [[ $dependency_stripped == \#* ]]; then
        continue
    # Skip blank lines
    elif [ -z "$dependency_stripped" ]; then
        continue
    else
        if pip install "$dependency_stripped"; then
            echo "$dependency_stripped is installed"
        else
            echo "Could not install $dependency_stripped, skipping"
        fi
    fi
done < recommends.txt