App Engine's URLFetch: http GET works locally

Question

The Problem:

Locally my application works fine. My HTTP GET returns code 200 and a final URL of: http://vow.mlspin.com/clients/index.aspx?

The rest of my application works great too.

When I deploy my code to GAE servers (I use the Eclipse plugin to do my deployment) my application ceases to work because the wrong html page is returned! the return code is still 200, but the URL is now (final URL): http://vow.mlspin.com/clients/signin.aspx?id=

My Question is: Is there a problem with redirects? Are the google app engine servers somehow blacklisted? What am I doing wrong here? Has anyone encountered this before?

The closest question I've found was this one: Http GET from GAE Java I've implemented this suggestion but it hasn't worked for me so far.

Thank you all in advance!

Additional Info -> below are HTTPResponse headers from the same exact GET request, one from the local deployment and the other from the GAE deployment.

Local HTTP Response Headers

Date :: Tue, 24 Apr 2012 04:12:32 GMT
Server :: Microsoft-IIS/6.0
X-Powered-By :: ASP.NET
X-AspNet-Version :: 2.0.50727
P3P :: CP="NOI DSP COR NID ADMa OPTa OUR NOR"
Cache-Control :: no-cache
Pragma :: no-cache
Expires :: -1
Content-Type :: text/html; charset=utf-8
Content-Length :: 133704

Deployed HTTP Response Headers

date :: Tue, 24 Apr 2012 04:11:19 GMT
server :: Microsoft-IIS/6.0
x-powered-by :: ASP.NET
x-aspnet-version :: 2.0.50727
p3p :: CP="NOI DSP COR NID ADMa OPTa OUR NOR"
cache-control :: private
content-type :: text/html; charset=utf-8
content-length :: 4991
x-google-cache-control :: remote-fetch
via :: HTTP/1.1 GWA

How I'm crafting my requests:

First I tried the easy way

Document doc = Jsoup.connect(baseMLSURL).get();

then I tried to go low level and just use java.net

private String getHttpFromServer(String url) throws IOException
    log.severe("getting http from: "+ url);
    StringBuilder sb = new StringBuilder();
    URL yahoo = new URL(url);
    URLConnection yc = yahoo.openConnection();
    yc.setRequestProperty("Host", "vow.mlspin.com");
    yc.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1");
    BufferedReader in = new BufferedReader(
                            new InputStreamReader(
                            yc.getInputStream()));
    String inputLine;

    while ((inputLine = in.readLine()) != null) {
        sb.append(inputLine.replaceAll("&nbsp", " ")+"\r\n");
    }
    in.close();

    return sb.toString();
}

Finally I've also tried to use Google's URLFetcher

private String getHttpUsingFetchService(String url) throws MalformedURLException, IOException {
    URLFetchService fetchService = URLFetchServiceFactory.getURLFetchService();
    HTTPResponse targetResponse = fetchService.fetch(new URL(url)); // Error
    log.severe("Code returned from request: "+targetResponse.getResponseCode());
    log.severe("final URL: "+targetResponse.getFinalUrl());
    String result = new String(targetResponse.getContent());
    return result.replaceAll(" ", " ");
}

Answer 1

The service you're trying to access requires authentication. Try visiting the first URL you posted in a new browser or an incognito window - you'll be redirected to the second URL. You need to perform the signin steps using your app, then take the cookies you're issued and send them on all subsequent requests.

Answer 2

We had something similar here a few months ago. In the end the mystery was that the site redirected to itself and expected to see some cookie it set back. But the urlfetch redirect handling doesn't send any cookies it receives. It's possible that the urlfetch emulation when running locally does something different with cookies.

If you can't get this to work, you may want to turn off redirect following in urlfetch and manage redirects and cookies yourself.