Installing Scrapy on Mac OSX 10.9.5

2020-07-22 17:44发布

问题:

I've new to python and have hit a wall with installing scrapy.

Environment Details:

MacBook pro

OSX 10.9.5

XCode and Command Line utilities are installed

Python 2.7.9 is installed in /usr/local/bin/python

Python 2.7.5 (distrib as part of OSX) is installed in /usr/bin/python

using pip install

Approach tried to date

Initial attempt to install (before adding the python 2.7.9) was impossible as pip was not present in the 2.7.5 default installation. Trying to add pip failed due to security violations, so I added the 2.7.9 installation.

I could then run pip, but hit problems with the fatal error (reported in other threads) due to libxml being missing, which was resolved by adding the command line tools.

The installation then ran further, but fails with the following error:

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_464d07d2xdb61d8fa.c -o build/temp.macosx-10.6-intel-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_464d07d2xdb61d8fa.o

src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_464d07d2xdb61d8fa.c:218:10: fatal error: 'CommonCrypto/CommonKeyDerivation.h' file not found

#include <CommonCrypto/CommonKeyDerivation.h>

         ^

1 error generated.

error: command '/usr/bin/clang' failed with exit status 1

Since hitting this error I have tried multiple approaches:

  • uninstall scrapy, and manually install cryptography - cryptography install fails with same error
  • uninstall scrapy and reinstall with pip command line option ==0.22 to install the prior version - install fails with same error
  • install using the pip command line option --user to install into local user directory - install fails with same error

As far as I can tell from searching, this issue has not been reported on Stackoverflow or Github. How can I get over this issue to successfully install scrapy? Thanks

回答1:

I managed to resolve this myself by a roundabout means, which I'm sharing as an answer to my own question.

I never got cryptography to install through pip. I got around this by installing anaconda. Cryptography installed fine through anaconda. I then successfully installed scrapy with pip install which recognized that cryptography was already present.