Beautiful Soup fetch dynamic table data

Question

I have the following code:

url = 'https://www.basketball-reference.com/leagues/NBA_2017_standings.html#all_expanded_standings'
html = urlopen(url)
soup = BeautifulSoup(html, 'lxml')

print(len(soup.findAll('table')))
print(soup.findAll('table'))

There are 6 tables on the webpage, but it only returns 4 tables. I tried to use 'html.parser' or 'html5lib' as parsers but did not work either.

Any idea how I can get the Table "expanded standings" from the webpage?

Thanks!

Answer 1

requests can't fetch data that are loaded by JS. So, you have to use selenium. First install selenium via pip - pip install selenium and download chrome driver and put the file in your working directory. Then try the following code.

from bs4 import BeautifulSoup
import time
from selenium import webdriver

url = "https://www.basketball-reference.com/leagues/NBA_2017_standings.html"
browser = webdriver.Chrome()

browser.get(url)
time.sleep(3)
html = browser.page_source
soup = BeautifulSoup(html, "lxml")

print(len(soup.find_all("table")))
print(soup.find("table", {"id": "expanded_standings"}))

browser.close()
browser.quit()

See selenium documentation.

If you are on Linux and get error Chromedriver executable needs to be in the PATH then try following these ways - link-1, link-2