Unable to install Python packages using pip in Ubu

Question

Previously I used to install packages by pip but now I am trying to install a Python library using pip, getting an SSL error:

 /home/teleduce/.virtualenvs/teleduce_handler/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
 /home/teleduce/.virtualenvs/teleduce_handler/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Could not fetch URL https://pypi.python.org/simple/xlwt/: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version - skipping

OpenSSL and TLS Version

OpenSSL 1.0.1 14 Mar 2012
SSLv3
TLSv1.2

Pip version

pip 8.1.2 from /home/teleduce/.virtualenvs/project_name/local/lib/python2.7/site-packages (python 2.7)

OS Information

Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-44-generic x86_64)

I tried

pip install --upgrade pip
curl https://bootstrap.pypa.io/get-pip.py | python

but it does not work for me. Got an error message is

SSL routines: SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

How do I fix this error?

Answer 1

The SSLError occurs because system OpenSSL library version (the one linked to your Python upon compilation) was below 1.0.1 the day when Python has been installed or your current Python version is below 2.7.9 / 3.4, because neither of these really support TLS 1.2 protocol version which the Python Package Index (PyPI) now requires from pip to connect.

Distributions usually cannot easily upgrade old openssl and system Python without undergoing a full OS upgrade, which is not always desirable. You could compile your own 'non-system' OpenSSL from recent sources and then try to compile a standalone 'non-system' Python linking it against the OpenSSL you have just compiled, but sometimes this approach is also unfeasible due to various limitations.

Solution

Popular recommendations, such as to pip install requests[secure] or urllib3[secure], often cannot help fix pip because pip itself is affected and won't be able to connect to PyPI to install anything. We cannot ask pip to connect to PyPI to fix pip's inability to connect to PyPI. :) To fix it without upgrading Python, we need to install relevant packages manually, resolving dependencies:

• PyOpenSSL and cryptography (its manylinux1 wheel ships newer openssl library);
• their dependencies: asn1crypto, cffi, enum34, idna, ipaddress, pycparser, six;
• any pip 10+ version, because older pip versions did not really use cryptography - only the standard library's ssl module (you don't require a new pip version if yours is already 10 or above, any pip v10+ will do)

Tested on ancient Ubuntu with old non-working pip and outdated system openssl version.

Step 1 - Download

Download the following packages from Python Packing Index (pypi.org) via your web browser of choice -- choose recent manylinux1 wheels (.whl) for your OS/platform:

pip, asn1crypto, enum34, idna, six, ipaddress, pyOpenSSL, cffi, cryptography wheels; and also pycparser (a non-wheel, it will be a tar.gz)

cp27- stands for Python 2.7, cp36- for Python 3.6;
mu- type manylinux wheels are a common choice, as they are for Pythons that store Unicode data in UCS-4 (UTF-32) format -- here's how to check it:
$ python -c "import sys; print('UCS4/UTF-32: mu-manylinux1' if sys.maxunicode > 65535 else 'UCS2/UTF-16: m-manylinux1')"

Note for Python 3: the cp34-abi3-manylinux1 cryptography's wheel can be used with any Python version>=3.4 because abi3 support multiple versions of Python3, e.g cryptography-2.5-cp34-abi3-manylinux1_x86_64.whl (2.4 MB)

Basically, wheels are ZIP archives with a specially formatted file name and the .whl extension, containing a relocatable Python package. The package can be pure-python, but also can have pre-compiled C libraries for python bindings, so it can be installed without the need to have certain system dependencies like gcc, python-dev and other C headers/libs, often required for classic .tar.gz format packages. This also allows to use exact versions of programs bundled within each wheel. The manylinux1_{x86_64,i686} wheel platform tag was adopted in PEP-513 and will work on many linux systems, including the popular desktop and server distros in common use. Expect manylinux2 tag in future!

Simply create a new directory, for example:
$ mkdir ~/wheels_dir
and copy (or move) all the downloaded packages to that directory.

No other files (except the downloaded wheels) and no subdirs there please!

Step 2 - Install

If your current pip version is below 8.1, the newer pip version has to be installed before proceeding with all other packages:
$ pip install --user --no-index ~/wheels_dir/pip-19.0.1-py2.py3-none-any.whl
It will upgrade pip to handle the new multilinux1 wheel format and help avoid the "not a supported wheel on this platform" error.

To install all the packages at user home level:
$ pip install --user --no-index ~/wheels_dir/*
$ pip3 in Python 3

If installing in a new or existing virtualenv, omit the --user option:

$ source bin/activate
$ pip install --no-index ~/wheels_dir/*

Pip will resolve correct installation order and dependencies automagically. (one could also create a requirements.txt for this if so needed)

Note: Unless you install in a Python virtualenv or venv, it is highly recommended to always use --user flag with pip. It then deploys python packages under your home dir in ~/.local/lib/ In fact, this option is always On by default in distro-patched pip versions provided by python3-pip and python-pip packages in recent versions of popular distros such as Ubuntu, Debian, Fedora, etc. Please try to avoid sudo pip, as using pip with root access interferes with your OS package manager subsystem (apt, yum, etc) and may affect essential OS components that depend on the distro-supplied system python.

Run $ pip freeze (or pip3 freeze in Python 3) command to check the results and ensure all packages have been installed for your Python environment.

Congratulations! Now your pip should work with PyPI, and you can try to look up something like pip search colorama from the online PyPI repo.

Verify

You can see the detailed summary of your system SSL/TLS setup by querying the installed pyOpenSSL lib directly:
$ python -m OpenSSL.debug
(a ModuleNotFoundError would mean the pyOpenSSL package was not installed)

Cryptography's linked OpenSSL shared lib doesn't conflict in any way with your system Python's openssl version. It may now be a good opportunity to also update your collection of root SSL certificates for the future by installing the latest python certifi package.

Why it works

Earlier versions of pip (before 10) only used the standard library's ssl module (which is a Python API to system OpenSSL library) without any possible fallback to other libraries like cryptography. Since version 10, pip now can use pyOpenSSL with cryptography, if present in the environment.

The manylinux1 wheel of cryptography package includes recent OpenSSL library that supports all TLS protocols as high as v1.3 regardless of what's on your platform (PyPI expects pip to support TLSv1.2). That's why this wheel weighs 2.1 Mb -- the archive ships a shared lib binding:

$ strings site-packages/cryptography/hazmat/bindings/_openssl.so | grep OpenSSL -m1  
OpenSSL 1.1.1a  20 Nov 2018  
$ python -c "from cryptography.hazmat.backends.openssl import backend as b; print b.openssl_version_text()"  
OpenSSL 1.1.1a  20 Nov 2018  
$ python -c "from OpenSSL import SSL; print SSL.SSLeay_version(0)"  
OpenSSL 1.1.1a  20 Nov 2018  
$ python -c "import requests; print requests.get('https://www.howsmyssl.com/a/check').json()['tls_version']"  
TLS 1.3  

The Cryptography wheel contains a statically-linked OpenSSL binding, which ensures that you have access to the most-recent OpenSSL releases without corrupting your system dependencies.
This will allow you to continue to use relatively old Linux distributions (such as LTS releases), while making sure you have the most recent OpenSSL available to your Python programs. (https://cryptography.io/en/latest/installation/)

In Python 2, the standard library's ssl module began supporting PROTOCOL_TLSv1_2 flag explicitly since version 2.7.9, while in Python 3 - since version 3.4; but TLSv1.2 connections would only work if and only if the TLSv1.2-capable system-wide OpenSSL library was already available in the system by the time Python was being compiled and linked against it. TLSv1.2 requires a minimum of OpenSSL 1.0.1 to function but OpenSSL 1.0.2 (or later) is generally recommended (it uses TLSv1.2 by default).

If you do have Python 2.7.9+ or 3.4+, and its ssl module had been, in fact, compiled against system openssl, say v1.0.2k, then even old pip (such as v6.0.8) would still be working with PyPI as of the time of this writing, and you would not even need cryptography for that. To check the standard library Python ssl and system openssl versions:
$ python -c "import ssl; print(ssl.OPENSSL_VERSION)" && openssl version
OpenSSL 0.9.8o 01 Jun 2010

Even if we upgraded some outdated distro-supplied openssl, or compiled the newest one, we can't just re-link the existing Python installation to it: the ssl module was hard-linked to the system-supplied OpenSSL upon compilation/installation of Python, and not vice versa. So, basically, one could not take advantage of new TLS protocols without recompiling/reinstalling Python itself (that should be versions 2.7.9+ / 3.4+ at least) to link it to the new system openssl library. This is where the above pyopenssl+cryptography approach comes to the rescue.

Happy TLSing! :)