How do I CURL www.google.com - it keeps redirectin

Question

I am using CURL to check for the existence of a URL (HEAD request) but when I test it with www.google.com, it redirects me to www.google.co.uk - probably because my server is UK-based.

Is there a way you can stop this from happening? I don't want to remove the CURLOPT_FOLLOWLOCATION option as this is useful for 301 redirects etc.

Part of my code is below;

$ch = curl_init();

    // set URL and other appropriate options
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
    curl_setopt($ch, CURLOPT_NOBODY, true);
    curl_setopt($ch, CURLOPT_AUTOREFERER, true);
    curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
    curl_setopt($ch, CURLOPT_TIMEOUT, 4);

    $output = curl_exec($ch);

    // get data     
$data = curl_getinfo($ch);

$data['url'] contains www.google.co.uk when I set $url as www.google.com

Answer 1

A bit of a hack, but how about using an IP address? http://216.239.59.147/ http://66.102.7.104/

Answer 2

You should turn off the follow location from curl (set it to false) and you won't be redirected anymore ...

   curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);

Answer 3

You could use www.google.co.uk directly, no difference there. google.com/.net always redirect to your location but if you use a country TLD like .co.uk it will not redirect.

There is no way (known to me) to prevent the redirect when using .com or .net.

Answer 4

Another option is to use simply encrypted.google.com. That won't redirect.

Answer 5

One way to avoid Google from deciding what country you are in, is by setting a different IP address. Just get one of the many US proxy servers from the Web and do something like this:

$ch=curl_init();
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_FOLLOWLOCTION,1); 
curl_setopt($ch,CURLOPT_PROXY,"8.12.33.159");
curl_setopt($ch,CURLOPT_PROXYPORT,"80");
curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3");
curl_setopt($ch,CURLOPT_URL,$URI);
$results=curl_exec($ch);
curl_close($ch);

This way, Google will think you come form a US IP address and not redirect to a local Google.

Answer 6

You need to use curl with a cookie that simulate a similar behavior in a browser.

When you visit google.com from England it redirects you to google.co.uk, however there is a link on that page titled "go to google.com" that lets you go back to google.com and stay there. It uses a cookie to remember your site preferences.

For example, here are the cookies that I have after doing this (using firefox):