I am trying to get the follower list of a given Twitter account. To do so, I manage to connect to my account using Selenium and PhantomJS (v2.1.1). Then, since Twitter is using infinite scrolling, I would like to implement the following algorithm:
1/ Count the number of html elements with class attribute equals ProfileCard-screennameLink
2/ Repeat:
-Scroll down
-Count the number of html elements with class attribute equals ProfileCard-screennameLink
Until the number of elements does not change
My problem is that the scrolling function does not work. Indeed, when I am running
elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
print("c", elemsCount)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(10)
elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
print("c (new)", elemsCount)
There is no change in the number of retrieved HTML elements (neither in the height of the page).. It looks like the scrollTo
function has no impact.
Do you have any explanation or workaround to solve my problem please?