installing Scrapy under Windows7

Question

I try to install Scrapy using easy install. I installed Cython and it seems to work, I installed libxml2 form http://www.lfd.uci.edu/~gohlke/pythonlibs/ I changed cygwinccompiler following this post scrapy installation error I still got an error and don't know what to do next. Also I don't know why there is that it is Building without Cython I have Windows 7 32-bit and Python 2.7

C:\Users\t>easy_install -U Scrapy
Searching for Scrapy
Reading http://pypi.python.org/simple/Scrapy/
Reading http://scrapy.org
Best match: scrapy 0.16.1
Processing scrapy-0.16.1-py2.7.egg
scrapy 0.16.1 is already the active version in easy-install.pth
Installing scrapy script to C:\Python27\Scripts
Installing scrapy.bat script to C:\Python27\Scripts

Using c:\python27\lib\site-packages\scrapy-0.16.1-py2.7.egg
Processing dependencies for Scrapy
Searching for lxml
Reading http://pypi.python.org/simple/lxml/
Reading http://codespeak.net/lxml
Best match: lxml 3.0.1
Downloading http://lxml.de/files/lxml-3.0.1.tgz
Processing lxml-3.0.1.tgz
Running lxml-3.0.1\setup.py -q bdist_egg --dist-dir c:\users\t\appdata\lo
cal\temp\easy_install-7mgdch\lxml-3.0.1\egg-dist-tmp-686xm8
Building lxml version 3.0.1.
Building without Cython.
ERROR: 'xslt-config' is not recognized as an internal or external command,
operable program or batch file.

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt
warning: no files found matching '*.txt' under directory 'src\lxml\tests'
In file included from src\lxml\lxml.etree.c:254:0:
c:\users\t\appdata\local\temp\easy_install-7mgdch\lxml-3.0.1\src\lxml\inc
ludes/etree_defs.h:9:31: fatal error: libxml/xmlversion.h: No such file or direc
tory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1

Answer 1

Not sure if you have solved it already. I ran into this problem today and here is how I solved it, grab the lxml binary directly: http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

and pip install Scrapy somehow ignores the fact that you have it installed and tries to build again, so it will fail. However, in the directory where you run pip install there will be a 'build' directory, find 'Scrapy\setup.py' there.

Now find at the bottom of the file:

setup_args['install_requires'] = ...

and remove lxml and pyopenssl (you will most likely need to install the binary directly if you are working on windows, which btw can be found here: http://slproweb.com/products/Win32OpenSSL.html)

run pip install Scrapy again from the same directory and you should be good to go.

Answer 2

I had the same problem (Windows 10, Python 2.7.12) and here's what worked for me. I used "pip install scrapy" instead of "easy_install". If you don't have pip installed for 2.7, here's the official documentation. If that still doesn't work you can try following these manual installation directions