Element inside tag not found back with Beau

Question

I have tried using Beautiful Soup with Python to crawl data from a website.

When I inspect the website itself I see the following:

"<span id="test"> 567 </span>"

but when I use Beautiful soup, all I see is:

"<span id="test"></span>"

It is as if the number has been hidden for security and protection purposes but how can I get this info from parsing? I think it is a dynamic JS element but I don’t know for sure and how to access it.

Answer 1

Here is an example code to implement it:

import scrapy
from selenium import webdriver
from bs4 import BeautifulSoup as bs

class SomeSpider(scrapy.Spider):
    name = "some"
    allowed_domains = ["yourdomain.com"]

    def __init__(self, *a, **kw):
        super(SomeSpider, self).__init__(*a, **kw)
        self.start_urls = ['http://www.yoururl.com',]
        firefox_profile = webdriver.FirefoxProfile()
        self.driver = webdriver.Firefox(firefox_profile=firefox_profile) 


   def parse(self, response):
       self.driver.get(response.url)
       page = TextResponse(response.url, body=self.driver.page_source, encoding='utf-8')  

      #HERE whatever you want do do eg. text = page.xpath('//span[@id="test"]').extract_first()