{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 混吃等死 的问题《Different web site response with RoboBrowser》','https://www.manongdao.com/q-673176.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
I'm trying to retrieve the HTML for a web site. When I view the page in Chrome I see a form with dates and rate options. But when I request the site with RoboBrowser I see a short HTML response which is completely different. What I am doing wrong?
My code:
from robobrowser import RoboBrowser
browser = RoboBrowser(parser='html.parser')
browser.open('http://www.marriott.com/reservation/availabilitySearch.mi?propertyCode=ATLRZ')
print(browser.parsed)
Result:
<!DOCTYPE doctype html>
<html>
<head><script src="/common/js/marriottCommon.js" type="text/javascript"> </script>
<meta charset="utf-8">
</meta></head>
<body>
<script>
var xhttp = new XMLHttpRequest();
xhttp.addEventListener("load", function(a,b,c){
window.location.reload()
});
xhttp.open('GET', '/reservation/availabilitySearch.mi?istl_enable=true&istl_data', true);
xhttp.send();
</script>
</body>
</html>
Try to imitate what a javascript-running browser would do. It looks like the site is reloading the page when it loads. That implies it might not let you access it unless you have the right cookies. The xhr request might be an additional check to prevent bots. So use
browser.sessionto keep the cookies, send the xhr a request and then try to reload the page with your new cookies.