Why Scrapy returns an Iframe?

2019-02-16 03:46发布

站内文章 / Python
19 0 0

问题:

i want to crawl this site by Python-Scrapy

i try this

class Parik(scrapy.Spider):
    name = "ooshop"
    allowed_domains = ["http://www.ooshop.com/courses-en-ligne/Home.aspx"]

    def __init__(self, idcrawl=None, proxy=None, *args, **kwargs):
        super(Parik, self).__init__(*args, **kwargs)
        self.start_urls = ['http://www.ooshop.com/courses-en-ligne/Home.aspx']

    def parse(self, response):
        print response.css('body').extract_first()

but i don't have the first page, i have an empty iframe

2016-09-06 19:09:24 [scrapy] DEBUG: Crawled (200) <GET http://www.ooshop.com/courses-en-ligne/Home.aspx> (referer: None)
<body>
<iframe style="display:none;visibility:hidden;" src="//content.incapsula.com/jsTest.html" id="gaIframe"></iframe>
</body>
2016-09-06 19:09:24 [scrapy] INFO: Closing spider (finished)

回答1:

The website is protected by Incapsula, a website security service. It's providing your "browser" with a challenge that it must perform before being given a special cookie that gives you access to the website itself.

Fortunately, it's not that hard to bypass. Install incapsula-cracker and install its downloader middleware:

DOWNLOADER_MIDDLEWARES = {
    'incapsula.IncapsulaMiddleware': 900
}


标签： python iframe web-scraping scrapy web-crawler
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~