Parse HTML Infinite Scroll

问题:

I'm trying to parse the HTML of a page with infinite scrolling. I want to load all of the content so that I can parse it all. I'm using Python. Any hints?

回答1:

Those pages update their html with AJAX. Usually you just need to find the new AJAX requests send by browser, guess the meaning of the AJAX url parameters and fetch the data from the API.

API servers may validate the user agent, referer, cookie, oauth_token ... of the AJAX request, keep an eye on them.

回答2:

the data is

1. either loaded in advance

2. or the page sends a request while you scroll

you can use httpfox to find the request and send it