Java HTTP getResponseCode returns 200 for non-exis

Question

I was expecting this code to return a 404, however it produces the output :

"Response code is 200"

Would it be possible to learn how to differentiate between existent and non-existent web pages . . . thanks so much,

    try
    {
    // create the HttpURLConnection
    URL url = new URL("http://www.thisurldoesnotexist");
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();

    System.out.println("Response code is " + connection.getResponseCode());
    }

Answer 1

You could:

1. Resolve the IP from the host of the page
2. Try to connect to port 80 on the resolved IP using plain sockets

This is a bit low level however and will add complexity since you will need to make a simple GET request through the socket. Then validate the response so you're sure that its actually a HTTP server running on port 80.

NMap might be able to help you here.

Answer 2

EDIT: I see you've call openConnection() but not connect() - could that be the problem? I would expect getResponseCode() to actually make the request if it hasn't already, but it's worth just trying that...

---

That suggests you've possible got some DNS resolver which redirects to a "helper" (spam) page, or something like that.

The easiest way to see exactly what's going on here is to use Wireshark - have that up and capturing traffic (HTTP-only, to make life easier) and then run your code. You should be able to see what's going on that way.

Note that I wouldn't have expected a 404 - because that would involve being able to find a web server to talk to to start with. If you're trying to go to a host which doesn't involve, there shouldn't be an HTTP response at all. I'd expect connect() to throw an exception.

Answer 3

Ideally you should be getting this error:

java.net.UnknownHostException: www.thisurldoesnotexist

But it looks like your URL is resolved by you DNS provider.

For instance on my company's network running your code with URI "http://profile/" displays the employee profile.

Please also check etc.home file if you are on windows to check if any settings have been changed.

Answer 4

try adding a "connection.connect();" or look at the contents returned...

it could be a dns issue, ie: your dns is being sent to a parking place... for example: freedns does this.

Answer 5

Like @spgennard - I think this is most likely a DNS issue.

• The URL you have chosen is owned by a DNS speculator.
• The URL you have chosen is "parked" by a DNS provider.
• Your ISP is messing with your DNS results to send your browser to some search page.

It is also possible that you are accessing the web via a proxy, and the proxy is doing something strange.

The way to diagnose this is to look at the other information in the HTTP responses you are getting, particularly the response body.