How do I prevent a 404/java.io.FileNotFoundExcepti

问题:

I need to fetch the full html of of a page produced by an Amazon Web Service. My question has to do with why I am unable to obtain this html from the AWS servers.

The AWS service provides an initial URL (URL #1) which is normally processed by a browser. URL #1 returns a Status of 302 and the URL of a redirected page (URL #2).

This works fine when a browser works with these URLs. However, when our Java code below runs we get a Status of 404 for URL #2 and a runtime error of: java.io.FileNotFoundException: "URL #2"

Here is the code:

URL url = null;
HttpsURLConnection conHTTPS = null;
int nResponseCode = -999;
try {
    // Load url with URL #1.
    url = new URL("URL #1");

    conHTTPS = (HttpsURLConnection)url.openConnection();                        // create a Connection object for this URL but we don't actually connect yet
    conHTTPS.setInstanceFollowRedirects(false);                                 // without this we get a Status = 404 in getResponseCode() below on this initial URL!
    conHTTPS.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");    // try to make us "look like a browser"
    conHTTPS.setRequestProperty("Accept","*/*");
    conHTTPS.setRequestMethod("GET");
    // Java 7 defaults to TLS 1.0 so must do this before getResponseCode() or will throw with javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    SSLContext ssl = null;
    try {
        ssl = SSLContext.getInstance("TLSv1.2");
        ssl.init(null,null,new SecureRandom());
        }
    catch (final NoSuchAlgorithmException | KeyManagementException e) {
        e.printStackTrace();
        }
    conHTTPS.setSSLSocketFactory(ssl.getSocketFactory());                       // get this connection setup for TLS 1.2
    conHTTPS.connect();                                                         // actually connect to URL #1 (should be done by getResponseCode() but we do it to be sure)
    nResponseCode = conHTTPS.getResponseCode();                                 // nResponseCode is returned as 302 (a redirect)
    }
catch (final MalformedURLException e1) {
    e1.printStackTrace();
    }
catch (final IOException e1) {
    e1.printStackTrace();
    }

// Use the response to URL #1 to get the redirection URL #2.
URL urlRedir = null;
HttpsURLConnection conHTTPSRedir = null;
try {
    // The prior conHTTPS.connect(); returned the redir URL in the "Location" header so get it.
    final URL urlBase = conHTTPS.getURL();
    final String sLocation = conHTTPS.getHeaderField("Location");
    urlRedir = new URL(urlBase,sLocation);                                      // use these two parts to build URL #2
    conHTTPSRedir = (HttpsURLConnection)urlRedir.openConnection();
    conHTTPSRedir.setInstanceFollowRedirects(false);
    try {
        sslRedir = SSLContext.getInstance("TLSv1.2");
        sslRedir.init(null,null,new SecureRandom());
        }
    catch (final NoSuchAlgorithmException | KeyManagementException e) {
        e.printStackTrace();
        }
    conHTTPSRedir.setSSLSocketFactory(sslRedir.getSocketFactory());
    conHTTPSRedir.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
    conHTTPSRedir.setRequestProperty("Accept","*/*");
    conHTTPSRedir.setRequestMethod("GET");
    conHTTPSRedir.connect();
    nResponseCode = conHTTPSRedir.getResponseCode();                        // nResponseCode is returned as 404!
    final InputStream inputStream = conHTTPSRedir.getInputStream();         // consistent with the 404, this throws java.io.FileNotFoundException
    }
catch (final IOException e1) {
    // Get the html of the error page the AWS server returns.
    final InputStream is = conHTTPSRedir.getErrorStream();
    final String contentEncoding = conHTTPSRedir.getContentEncoding() != null ? conHTTPSRedir.getContentEncoding() : "UTF-8";
    final String sErrorPage = IOUtils.toString(is, contentEncoding); //Apache Commons IO
    }

Here is the (only) informative part of what sErrorPage from the catch block above contains:

The page you tried was not found.

You may have typed the address incorrectly or you may have used an outdated link.

If I use the debugger and step through the code and stop immediately after I obtain urlRedir and manually take its value and paste it into the browser we see the page as we should, so this URL #2 is good and works in the browser. But if I had let the code continue we get a 404 for that same URL!

Can anyone explain what is wrong here?

Thank you.