Java jconsole jmx connection failure

Question

I'm trying to connect jconsole to a jvm invoked by:

java \
-Djava.util.logging.config.file=./logging.properties \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=12700 \
-cp . Sleep

I then try to start jconsole with:

jconsole -J-Djava.util.logging.config.file=./logging.properties

The loggin.properties file includes:

sun.rmi.level=FINEST

After opening a socket to port 12700, rmi then appears to attempt a connection on another port :

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.40.243.12:12700/jmxrmi] connecting...
Jan 5, 2012 2:30:42 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://10.40.243.12:12700/jmxrmi] finding stub...
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint <clinit>
FINE: JConsole.addHost: localHostKnown = true, localHost = 10.206.6.59
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPTransport <init>
FINE: JConsole.addHost: Version = 2, ep = [10.206.6.59:0]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
FINE: JConsole.addHost: created local endpoint for socket factory null on port 0
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [10.40.243.12:12700]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: 10.40.243.12, port: 12700
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: server suggested 10.206.6.59:12306
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINER: JConsole.addHost: using 10.206.6.59:0
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINER: JConsole.addHost: create call context
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef logClientCall
FINER: JConsole.addHost: outbound call: [endpoint:[10.40.243.12:12700](remote),objID:[0:0:0, 0]] : sun.rmi.registry.RegistryImpl_Stub[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall <init>
FINER: JConsole.addHost: write remote call header...
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall getOutputStream
FINER: JConsole.addHost: getting output stream
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef invoke
FINER: JConsole.addHost: execute call
Jan 5, 2012 2:30:43 PM sun.rmi.transport.StreamRemoteCall getInputStream
FINER: JConsole.addHost: getting input stream
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "javax.management.remote.rmi.RMIServerImpl_Stub", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "javax.management.remote.rmi.RMIServerImpl_Stub" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "java.rmi.server.RemoteStub", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "java.rmi.server.RemoteStub" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINE: JConsole.addHost: name = "java.rmi.server.RemoteObject", codebase = "", defaultLoader = sun.misc.Launcher$AppClassLoader@a39137
Jan 5, 2012 2:30:43 PM sun.rmi.server.LoaderHandler loadClass
FINER: JConsole.addHost: class "java.rmi.server.RemoteObject" found via defaultLoader, defined by null
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef done
FINE: JConsole.addHost: free connection (reuse = true)
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: reuse connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel free
FINE: JConsole.addHost: create reaper
Jan 5, 2012 2:30:43 PM sun.rmi.server.UnicastRef newCall
FINE: JConsole.addHost: get connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPChannel createConnection
FINE: JConsole.addHost: create connection
Jan 5, 2012 2:30:43 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
FINER: JConsole.addHost: opening socket to [tdiap12:41096]
Jan 5, 2012 2:30:43 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
FINE: JConsole.addHost: host: tdiap12, port: 41096

The first connection attempt succeeds because on the remote server, I can see ESTABLISHED connection.

wse2tst@tdiap12:~> netstat -a | grep 12700
tcp        0      0 *:12700                     *:*                         LISTEN      
tcp        0      0 tdiap12.vgcar.net:12700     per-00c0016253a2.vgca:12252 ESTABLISHED 

The packet trace indicates a successful connection as well.

The second connection to port 41096 times out, and the jconsole application reports a "Connection failure" and terminates the first connection as well. Why is a second connection being attempted? Is there any way to specify the port for this second connection? The target server is tightly controlled, and other ports are blocked by firewall rules. On multiple connection attempts, the second port will change to different random values.

Thanks for any help, Steve

Answer 1

You can set the second port with parameter -"Dcom.sun.management.jmxremote.rmi.port" where port number can be same as that of first one . Even after setting this it will promt for insecure connection ... please allow insecure connection then only it will allow you to proceed further otherwise till then in log you will find error failed to connect: java.lang.SecurityException: Expecting a sun.rmi.server.UnicastRef2 remote reference in stub!

Thanks

Answer 2

The out-of-the-box JMX implementation uses two ports - one for the registry (the one that you specified) and one for the actual connection which is selected randomly (!). This is quite a design flaw since the random selection of the second port makes it very hard to configure a firewall.

There are ways around it however - either you can do it yourself manually, or if you are using Tomcat you let it handle it for you.

Answer 3

make sure you run your application using below command,

java -Dcom.sun.management.jmxremote.port=9595 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -jar <your jar file name>

then open jconsole with admin privilege; you might be able to connect without any issue.

if you want to run multiple jar files, make sure to use different ports; otherwise it will give error saying it's already bind by the JVM

Answer 4

new and better solution than using a manually coded agent, is to use the new parameter which seems to be introduced in Java 7:

-Dcom.sun.management.jmxremote.rmi.port=7091

So in combination - same port can be used:

-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.rmi.port=7091

Solution found here: http://hirt.se/blog/?p=289