Setting user agent of a java URLConnection

问题:

I\'m trying to parse a webpage using Java with URLConnection. I try to set up the user-agent like this:

java.net.URLConnection c = url.openConnection();
c.setRequestProperty(\"User-Agent\", \"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2\");

But the resulting user agent is the one I specify, with \"Java/1.5.0_19\" appended to the end. Is there a way to truly set the user agent without this addition?

回答1:

Off hand, setting the http.agent system property to \"\" might do the trick (I don\'t have the code in front of me).

You might get away with:

 System.setProperty(\"http.agent\", \"\");

but that might require a race between you and initialisation of the URL protocol handler, if it caches the value at startup (actually, I don\'t think it does).

The property can also be set through JNLP files (available to applets from 6u10) and on the command line:

-Dhttp.agent=

Or for wrapper commands:

-J-Dhttp.agent=

回答2:

Just for clarification: setRequestProperty works just fine! At least with Java 1.6.30.

I listened on my machine with netcat(a port listener):

$ nc -l -p 8080

It simply listens on the port, so you see anything which gets requested, like raw http-headers.

And got the following http-headers without setRequestProperty:

GET /foobar HTTP/1.1
User-Agent: Java/1.6.0_30
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

And WITH setRequestProperty:

GET /foobar HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

As you can see the user agent was properly set.

Full example:

import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;


public class TestUrlOpener {

    public static void main(String[] args) throws IOException {
        URL url = new URL(\"http://localhost:8080/foobar\");
        URLConnection hc = url.openConnection();
        hc.setRequestProperty(\"User-Agent\", \"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2\");

        System.out.println(hc.getContentType());
    }

}

回答3:

Slightly changed Tom Hawtins answer to:

 System.setProperty(\"http.agent\", \"\"); 

according to http://www.ivoa.net/forum/apps/0903/0610.htm

回答4:

its work for me set the User-Agent in the addRequestProperty.

URL url = new URL(<URL>);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.addRequestProperty(\"User-Agent\",\"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0\");

回答5:

HTTP Servers tend to reject old browsers and systems.

The page Tech Blog (wh): Most Common User Agents reflects the user-agent property of your current browser in section \"Your user agent is:\", which can be applied to set the request property \"User-Agent\" of a java.net.URLConnection or the system property \"http.agent\".