PHP file_get_html is not working

问题:

I have used simple_html_dom library but i can not get HTML content only for 1 URL but i am getting 503 error. Check my below code.

$base = 'http://www.amazon.com/gp/offer-listing/B001F0M4K8/ref=dp_olp_all_mbc/183-8463780-9861412?ie=UTF8&condition=new';

echo $html = file_get_html($base);

Error : Warning: file_get_contents(http://www.amazon.com/gp/offer-listing/B001F0M4K8/ref=dp_olp_all_mbc/183-8463780-9861412?ie=UTF8&condition=new) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 503 Service Unavailable in D:\xampp\htdocs\webcrawler-amazon\webcrawler-amazon\simple_html_dom.php on line 76

I am stuck here so please help me.

回答1:

I think, server just blocks your request, you will not be able to fetch data from it, using simple HTTP requests.

You can try using curl, proxies, or both (there are ready to use solutions for this, like: AngryCurl, or RollingCurl)

回答2:

I recommand you to do this with cURL : http://php.net/manual/en/book.curl.php

You can use it with PHP or in command line. There is tons of example online.

回答3:

It's the Amazon's anti-bots defense system.

The returned page starts with the following HTML comment:

<!--
        To discuss automated access to Amazon data please contact [e-mail].
        For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_5_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_5_ac for advertising use cases.
-->

You need to either mimic very well the behaviour of a real customer using a browser or ask them about an approved way to get data from their systems automatically. Using an API is better (and easier) than scrapping web pages, anyway.

回答4:

I am doing the same, they are sending the following to you. Sometimes, you can get by it.

Enter the characters you see below
Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.
Type the characters you see in this image:
 

Try different image 
Continue shopping 

Conditions of Use Privacy Policy 
© 1996-2014, Amazon.com, Inc. or its affiliates