Scrapping of the Twitter follower page using selen

2019-05-28 18:12发布

问题:

I am trying to get the follower list of a given Twitter account. To do so, I manage to connect to my account using Selenium and PhantomJS (v2.1.1). Then, since Twitter is using infinite scrolling, I would like to implement the following algorithm:

1/ Count the number of html elements with class attribute equals ProfileCard-screennameLink 

2/ Repeat:
    -Scroll down
    -Count the number of html elements with class attribute equals ProfileCard-screennameLink
Until the number of elements does not change

My problem is that the scrolling function does not work. Indeed, when I am running

    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c", elemsCount)

    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(10)
    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c (new)", elemsCount)

There is no change in the number of retrieved HTML elements (neither in the height of the page).. It looks like the scrollTo function has no impact. Do you have any explanation or workaround to solve my problem please?