parse html content by passing custom date input

问题:

I am parsing data from here. On the webpage I can get data for example yesterday by selecting the desired date. How can I parse to get the same data (ie. yesterday)? Like, pass custom dates to get data for that date.

回答1:

You can either use Selenium or use the site's ajax api.
Here is an example of the latter:

def get_by_date(date):
    url = 'https://markets.ft.com/data/world/ajax/getnextecoevents?startDate=' + date
    r = requests.get(url)
    return r.json()['html']

date should be formatted as yyyy-mm-dd, eg: "2017-07-20"

Using the above function and bs4 to scrape the table contents:

html = get_by_date('2017-06-20')
soup = BeautifulSoup(html, 'html.parser')
data = [[td.text for td in tr.find_all('td')] for tr in soup.find('table').find_all('tr')]