{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 Deceive 欺骗 的问题《How to scrape this webpage with Python and lxml? e》','https://www.manongdao.com/q-507223.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
For educational purposes, I'm trying to scrape this page gradually with Python and lxml, starting with movie names.
From what I've read so far from the Python docs on lxml and the W3Schools on XPath, this code should yield me all movie titles in a list:
from lxml import html
import requests
page = requests.get('http://www.rottentomatoes.com/browse/dvd-top-rentals/')
tree = html.fromstring(page.text)
movies = tree.xpath('//h3[@class="movieTitle"]/text()')
print movies
Basically, it should give me every h3 element anywhere in the document that has the attribute class that has the value "movieTitle". Upon running the code though, I only get an empty list printed out.
I can't figure out why.
I tried by myself, so I ran:
movies = tree.xpath('//h3[@class]/text()')
print movies
Well this one should return any H3 with the attribute class, but it returns this list instead:
['From RT Users Like You!', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']
I tried targeting the first string on this list by targeting its class value ("noSpacing center"), and it returned this sole string successfully. So I'm sure there's something that I misunderstand about lxml/XPath works. Can anyone point me in helpful direction? Thanks in advance!
Information on http://www.rottentomatoes.com/browse/dvd-top-rentals/ is not rendered directly into the page but loaded from XMLHttpRequests.
The API you are looking for, seems to be:
http://d3biamo577v4eu.cloudfront.net/api/private/v1.0/m/list/find?page=1&limit=30&type=dvd-top-rentals&services=amazon%3Bamazon_prime%3Bflixster%3Bhbo_go%3Bitunes%3Bnetflix_iw%3Bvudu&sortBy=popularity
And the query string is prepared depending on the selected filters.
So you must make requests to that endpoint (instead of the URL you are currently requesting) and parse JSON to extract desired data.
You should play with "page" GET variable to get the next ones.
Example with cURL + jq:
Example with Python + Requests:
Output:
Using
seleniumis a another way to wait till the page is fully loaded (i.e. including allJavaScriptmanipulation). You don't have to useFirefox, you can use other browsers or a headless browser likePhantom JSif displaying the actual site is not required.Output: