How can I extract a javascript value in scrapy

Question

I am using scrapy to crawl youtube videos and I need the language of title/description of the video.When I use browser view source on this video I can inside a script tag there is a variable 'METADATA_LANGUAGE': 'no'. Can I extract this value in scrapy and its extensions or I should download and parse html with libraries like beautifulsoup / htmlparser.

Answer 1

Yes this is possible using Scrapy. You could take a look at this question.

There are many ways to achieve what you're looking for. One is to get the <script> tag using scrapy's selectors and then use regex to get the specific METADATA_LANGUAGE variable you're looking for.

Answer 2

Based on this you can select the text of script with xpath/css and then use regex to search the variable name. Assum the first script contains the METADATA_LANGUAGE:

items = response.xpath('//script/text()')[0].re(".*METADATA_LANGUAGE.*")