java.io.IOException: Server returned HTTP response

Question

This question already has an answer here:

• 403 Forbidden with Java but not web browser? 4 answers

I want to open a link from url : "http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular", sometimes i get:

java.io.IOException: Server returned HTTP response code: 403 for URL. But it's ok when open the url using browser. Below is part of my code:

URL url = new URL("http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular");

InputStream is = url.openConnection().getInputStream();

error detail

Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.kohls.com/search.jsp?N=0&search=jacket&WS=96 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) at Links.main(Links.java:41)

Answer 1

The particular webserver you are trying to access is checking the User-Agent HTTP header and denying access to anything that doesn't look like a normal browser, to prevent bots (which is probably what you are writing).

You just need to set the header as part of your request in Java and it will work.

How you set the header will depend on how you are making the connection, but if you are using a simple URLConnection then this will work:

URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0");

Normally a "real" User-Agent contains lots of extra info, but that webserver seems to look only for the basic browser type.

You can prove this using wget with and without the -U User-Agent option:

$ wget "http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular"
--2015-05-07 16:08:46--  http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular
2015-05-07 16:08:46 ERROR 403: Forbidden.

$ wget -U "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0" "http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular"
--2015-05-07 16:08:49--  http://www.kohls.com/search.jsp?search=jacket&submit-search=web-regular
awaiting response... 200 OK
...