Installing Scrapy on ubuntu 14.04 fails

Question

I'm getting error installing Scrapy on my ubuntu box. I'm using pip to install Scrapy. I'm aware that it needs setuptools to be installed. I got that installed using the script provided in setuptools website.

reading manifest file 'Twisted.egg-info/SOURCES.txt'

writing manifest file 'Twisted.egg-info/SOURCES.txt'

creating build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport

copying twisted/internet/iocpreactor/iocpsupport/iocpsupport.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport

copying twisted/internet/iocpreactor/iocpsupport/winsock_pointers.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport

copying twisted/test/raiser.c -> build/lib.linux-x86_64-2.7/twisted/test

copying twisted/runner/portmap.c -> build/lib.linux-x86_64-2.7/twisted/runner

copying twisted/python/sendmsg.c -> build/lib.linux-x86_64-2.7/twisted/python

running build_ext

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c conftest.c -o conftest.o

building 'twisted.runner.portmap' extension

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/twisted

creating build/temp.linux-x86_64-2.7/twisted/runner

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c twisted/runner/portmap.c -o build/temp.linux-x86_64-2.7/twisted/runner/portmap.o

twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directory

 #include <Python.h>

                    ^

compilation terminated.

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/Twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4QNuNV-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/Twisted
Storing debug log for failure in /root/.pip/pip.log

Any Idea where the process goes wrong? I've got gcc and g++ installed already.

Answer 1

From the error fatal error: Python.h: No such file or directory it looks like python development headers are not installed. Try this command and then try to install again.

sudo apt-get install python-dev

For installing libevent library apply this command,

sudo apt-get install libevent-dev

Answer 2

I figured this out on ubuntu 14.04 by (A) actually following the scrapy docs for installing on ubuntu, then (B) an error that a lot of others were getting, by following a S/O solution Error while starting new scrapy project

(A) http://doc.scrapy.org/en/1.0/topics/ubuntu.html#topics-ubuntu

then

(B)

sudo pip install pyasn1 --upgrade

Answer 3

Right after installing Fedora 23 that's what I did (may works on Ubuntu):

[root@x ~]# pip install scrapy
[root@x ~]# dnf install python-cffi
[root@x ~]# dnf install openssl-devel
[root@x ~]# dnf install gcc
[root@x ~]# dnf install redhat-rpm-config
[root@x ~]# dnf install libxml
[root@x ~]# dnf install libxml2-devel
[root@x ~]# dnf install libxml-devel
[root@x ~]# dnf install glib2-devel gnet2-devel
[root@x ~]# dnf install libxslt-devel

Answer 4

sudo apt install python3-scrapy

I have tried this ..and it worked for me. (You should have the ssh installed)