How do you open a file stream for reading using Sc

Question

Using Scrapy, I want to use my extracted url to read a binary file into memory and extract the contents.

Currently, I can find the URL on the page using a selector e.g.

myFile = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract()

How do I then read that file into memory so that I can look for content in that file?

Many thanks

Answer 1

Make a request and explore the content in the callback:

def parse(self, response):
    url = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract_first()
    return scrapy.Request(url, callback=self.parse_file)

def parse_file(self, response):
    # response here is the contents of the file
    print(response.body)