Scrapy - ValueError: Missing scheme in request url

Question

I'm getting the following traceback but unsure how to refactor.

ValueError: Missing scheme in request url: #mw-head

Full code:

class MissleSpiderBio(scrapy.Spider): 

    name = 'missle_spider_bio'
    allowed_domains = ['en.wikipedia.org']
    start_urls = ['https://en.wikipedia.org/wiki/...']

this is the part giving me issues (I believe)

    def parse(self, response):
        filename = response.url.split('/')[-1]
        table = response.xpath('///div/table[2]/tbody')
        rows = table.xpath('//tr')
        row = rows[2]
        row.xpath('td//text()')[0].extract()
        wdata = {}
        for row in response.xpath('//* \
        [@class="wikitable"]//tbody//tr'):
            for link in response.xpath('//a/@href'):
                link = link.extract()
                if((link.strip() != '')):
                    yield Request(link, callback=self.parse)
                    #wdata.append(link)
                else:
                    yield None
                #wdata = {}
                #wdata['link'] = BASE_URL + 
                #row.xpath('a/@href').extract() #[0]
                wdata['link'] = BASE_URL + link 
                request = scrapy.Request(wdata['link'],\
                callback=self.get_mini_bio, dont_filter=True) 
                request.meta['item'] = MissleItem(**wdata)
                yield request

here is the second part of the code:

    def get_mini_bio(self, response):
        BASE_URL_ESCAPED = 'http:\/\/en.wikipedia.org'
        item = response.meta['item']
        item['image_urls'] = [] 
        img_src = response.xpath('//table[contains(@class, \ 
        "infobox")]//img/@src')
        if img_src:
            item['image_urls'] = ['http:' + img_src[0].extract()]
        mini_bio = ''
        paras = response.xpath('//*[@id="mw-content-text"]/p[text()\ 
        or  normalize-space(.)=""]').extract()
        for p in paras:
            if p =='<p></p>':
                break
            mini_bio += p

        mini_bio = mini_bio.replace('href="/wiki', 'href="' + \ 
        BASE_URL + '/wiki')
        mini_bio = mini_bio.replace('href="#', item['link'] + '#')
        item['mini_bio'] = mini_bio
        yield item 

I tried refactoring but am now getting a:

ValueError: Missing scheme in request url: #mw-head

any help would be immensely appreciated

Answer 1

Looks like you were on the right track with the commented out [0].

xpath().extract() #returns a list of strings

You need to select the string with [0]

Answer 2

row.xpath('a/@href').extract()

That expression evaluates to a list NOT a string. When you pass the URL to the request object, scrapy expects a string, not a list

To fix this, you have a few options: You can use LinkExtractors which will allow you to search a page for links and automatically create scrapy request objects for those links:

https://doc.scrapy.org/en/latest/topics/link-extractors.html

OR You could run a for loop and go through each of the links:

from scrapy.spiders import Request

for link in response.xpath('//a/@href'):
    link = link.extract()
    if((link.strip() != '')):
        yield Request(link, callback=self.parse)
    else:
        yield None

You can add whatever string filters you want to that code

OR

If you just want the first link, you can use .extract_first() instead of .extract()