Splinter or Selenium: Can we get current html page

Question

I'm trying to crawl the website "http://everydayhealth.com". However, I found that the page will dynamically rendered. So, when I click the button "More", some new news will be shown. However, using splinter to click the button doesn't let "browser.html" automatically changes to the current html content. Is there a way to let it get newest html source, using either splinter or selenium? My code in splinter is as follows:

import requests
from bs4 import BeautifulSoup
from splinter import Browser

browser = Browser()
browser.visit('http://everydayhealth.com')
browser.click_link_by_text("More")

print(browser.html)

---

Based on @Louis's answer, I rewrote the program as follows:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Firefox()
driver.get("http://www.everydayhealth.com")
more_xpath = '//a[@class="btn-more"]'
more_btn = WebDriverWait(driver, 10).until(lambda driver: driver.find_element_by_xpath(more_xpath))
more_btn.click()
more_news_xpath = '(//a[@href="http://www.everydayhealth.com/recipe-rehab/5-herbs-and-spices-to-intensify-flavor.aspx"])[2]'
WebDriverWait(driver, 5).until(lambda driver: driver.find_element_by_xpath(more_news_xpath))

print(driver.execute_script("return document.documentElement.outerHTML;"))
driver.quit()

However, in the output text, I still couldn't find the text in the updated page. For example, when I search "Is Milk Your Friend or Foe?", it still returns nothing. What's the problem?

Answer 1

With Selenium, assuming that driver is your initialized WebDriver object, this will give you the HTML that corresponds to the state of the DOM at the time you make the call:

driver.execute_script("return document.documentElement.outerHTML;")

The return value is a string so you could do:

print(driver.execute_script("return document.documentElement.outerHTML;"))

Answer 2

When I use Selenium for tasks like this, I know browser.page_source does get updated.