Efficient method to scroll though pages using Sele

2020-02-26 12:58发布

问题:

I'm currently hard coding a scrolling using Selenium, Firefox Driver in Python, is there anyway for it to scroll smoothly up and down the page? The page can grow 20 times bigger than normal as the images in the page loads.

I need to scroll up and down multiple times , not missing out any part of the page.

The current method isnt efficient and sometimes doesnt scroll to some part of the page. The following is my current code

 browser = webdriver.Firefox()
browser.get(link)

browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.4);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.5);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.8);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.2);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.3);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.5);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.9);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.2);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.1);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.8);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/50);")
        time.sleep(0.2)

回答1:

You can use this if using Selenium in Python...

scheight = .1
while scheight < 9.9:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight/%s);" % scheight)
    scheight += .01

Or something along those lines. You can add if brakes to capture data at specific scheight locations within the while. Just make the += or iteration whatever you like. I used .01 so I could visually see it scroll, otherwise you would need timeouts and it would appear choppy. Depends on what you need.



回答2:

Can you try to just do all the scrolling in one shot?

 browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.5);window.scrollTo(0, document.body.scrollHeight/3.7);")

Just list out all the scroll calls in the same execute script call.