pip fails with AttributeError: 'module' ob

I recently found that pip was completely broken for me. This is on Ubuntu bionic 18.04.

$ pip --version    
Traceback (most recent call last):                                           
  File "/usr/bin/pip", line 9, in <module>
from pip import main                                   
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 22, in <module>
from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
  File "/usr/lib/python2.7/dist-packages/pip/_vendor/__init__.py", line 64, in <module>
vendored("cachecontrol")                      
  File "/usr/lib/python2.7/dist-packages/pip/_vendor/__init__.py", line 36, in vendored
__import__(modulename, globals(), locals(), level=0)
File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 4, in <module>
File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/__init__.py", line 84, in <module>
File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
 from OpenSSL import crypto, SSL                                                
  File "/usr/lib/python2.7/dist-packages/OpenSSL/crypto.py", line 12, in <module>
from cryptography import x509                                  
  File "/usr/lib/python2.7/dist-packages/cryptography/x509/__init__.py", line 8, in <module>
from cryptography.x509.base import (                       
  File "/usr/lib/python2.7/dist-packages/cryptography/x509/base.py", line 16, in <module>
from cryptography.x509.extensions import Extension, ExtensionType
  File "/usr/lib/python2.7/dist-packages/cryptography/x509/extensions.py", line 18, in <module>
from cryptography.hazmat.primitives import constant_time, serialization
  File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py", line 9, in <module>
from cryptography.hazmat.bindings._constant_time import lib
AttributeError: 'module' object has no attribute '_init_cffi_1_0_external_module'

I've seen other references but they don't match my situation, or don't have answers:

Python Cryptography run time error on _init_cffi_1_0_external_module - Stack Overflow
Bug #1512792 “Pip crashes with cryptography/cffi/openssl error” : Bugs : python-pip package : Ubuntu
Bug #1484870 “Devstack not working in Fedora 22 (python-cffi too...” : Bugs : devstack

How can I fix this?

After looking at this some more, I discovered that the Fedora bug report above had information that really helped:

at some point, cffi 1.1.2 from pip is being overwritten with cffi 0.8.6 from the python-cffi package. As cryptography 1.1.2 is making a call to the cffi_1_0_external_module' it doesn't exists and thus fails.

I dug around, and found that I had an obsolete cffi version in ~/.local/lib/python2.7/site-packages/ (presumably from a pip install --user ... some years ago).

A general way to explore that and check versions is:

>>> import cffi
>>> cffi.__version__
'0.8.6'
>>> cffi
<module 'cffi' from '$HOME/.local/lib/python2.7/site-packages/cffi/__init__.pyc'>

Removing that directory fixed pip, but presumably could have interefered with something else I installed long ago.

I also hear that it can help in some related situations to use easy_install (since pip isn't working...) to upgrade cffi:

easy_install -U cffi

I'm still curious if there is a large lesson here about how to avoid this sort of thing in the future.

Why is cffi both within pip and an external package?
Are local installs dangerous, since they persist past OS and Python upgrades?
Are there general packaging/dependency best practices that would avoid these sorts of package dependency problems?