Cassandra NoHostAvailableException Java-CQLDriver

问题:

I am building an application in java using Cassandra's CQL driver. I am able to run it properly when I use local cassandra server. However the same code when I try to run is not working for remote cassandra server. It gives the following error during initialization :-

Following are the maven dependecy that I am using for CQL driver:-

<dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-core</artifactId>
        <version>2.1.3</version>
    </dependency>

    <dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-mapping</artifactId>
        <version>2.1.2</version>
    </dependency>

Following is the error:-

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: yyyy.yyy.yyy.yyy.yyy-yyy.net/yy.yy.yy.yy:9160 (com.datastax.driver.core.ConnectionException: [yyyy.yyy.yyy.yyy.yyy-yyy.net/yy.yy.yy.yy:9160] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [yyyy.yyy.yyy.yyy.yyy-yyy.net/yy.yy.yy.yy:9160] Operation timed out)), 
    Unexpected error during transport initialization      (com.datastax.driver.core.ConnectionException: [xxxx.xxx.xxx.xxx.xxx-xxx.net/xx.xx.xx.xx:9160] Operation timed out)))
         at   com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)
    at com.datastax.driver.core.Cluster.init(Cluster.java:158)
    at com.datastax.driver.core.Cluster.connect(Cluster.java:246)

The local cassandra server on which everything works fine is:-

cqlsh 4.1.1, Cassandra 2.0.7 , CQL spec: 3.1.1, Thrift Protocol 19.39.0

The remote cassandra server on which connection is getting timed out is:-

cqlsh 3.1.8 | Cassandra 1.2.12 | CQL spec 3.0.0 | Thrift protocol 19.36.1

I am using the following code to connect. Can someone comment what is going wrong in the remote connection case?

clusterOBJECT = Cluster.builder().addContactPointsWithPorts(addrs)
                .withQueryOptions(new     QueryOptions().setConsistencyLevel(ConsistencyLevel.ONE))
                .withProtocolVersion(1).withoutJMXReporting()
                .withPort(port)
                .withSocketOptions(options)
                .withRetryPolicy(DefaultRetryPolicy.INSTANCE)
                .withLoadBalancingPolicy(new TokenAwarePolicy(new DCAwareRoundRobinPolicy())).build();
        session = clusterOBJECT.connect();

I am providing valid hostname along with port(9160) in addContactPointsWithPorts. Also the hostnames are pingable.

PS:- The following properties are already set in cassandra.yaml.

start_native_transport: true
native_transport_port: 9042

[EDIT] After changing the port to 9042, I was able to connect to Cassandra. BUT I see that the connection is getting dropped by this exception soon after.

com.datastax.driver.core.TransportException: [/xx.xx.xxx.xx:9042] Cannot connect
    at com.datastax.driver.core.Connection.<init>(Connection.java:106)
    at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:35)
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:528)
    at com.datastax.driver.core.DynamicConnectionPool.<init>(DynamicConnectionPool.java:74)
    at com.datastax.driver.core.HostConnectionPool.newInstance(HostConnectionPool.java:33)
    at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:270)
    at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:39)
    at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:303)
    at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:295)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
 Caused by: java.net.ConnectException: Connection refused: /xx.xx.xxx.xx:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    ... 3 more

Thanks.

回答1:

"I am providing valid hostname along with port(9160) in addContactPointsWithPorts. Also the hostnames are pingable."

After reading this line in your original post, and double checking some documentation, I think I know what is going on here. The About the Java Driver document states clearly that:

The Java Driver 2.0 for Apache Cassandra works exclusively with the Cassandra Query Language version 3 (CQL3) and Cassandra's new binary protocol

The connection requirements section also indicates:

you have configured the following in the cassandra.yaml:

start_native_transport : true
rpc_address : IP address or hostname

reachable from the client machines in the cluster can accept connections on port 9042. Note: The client port can be configured using the native_transport_port in cassandra.yaml.

In other words, you should not use Thrift (9160) to connect to Cassandra with the Java-CQL Driver. Try it without setting the port (in withPort or with the hostname(s)). It should try to use 9042 on its own.

Edit: Ok, I just tried it on my own using .addContactPoints (without ports specified with the hostnames) and .withPort(9160), and it worked. That would seem to contradict the documentation that I had referenced earlier in my post. In any case, I would still recommend using the native protocol (9042) as the Thrift protocol is being deprecated. Also, newer versions of Cassandra (2.1) and the DataStax CQL drivers have been shown to out-perform the Thrift-based drivers.

回答2:

I think that the key here is "local" and "remote". I've asked the question, but did not receive an answer, but I suspect that there is one of 2 issues:

1. Firewall is blocking ports 9160 and 9042 from the client. Have you tried to telnet to to the nodes to the ports 9160 and 9042?

2. There is a private/public address issue, that is usually observed when the cluster is inside of the private network, and the client trying to access the cluster using external IP addresses. In this case cluster has no idea about the public IPs, which are assigned by an infrastructure. In this case it is driver's responsibility to do this address translation. For example, Java driver has a policy called AddressTranslater that you can implement to perform translation from internal IP to external - for example by using AWS API. http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/policies/AddressTranslater.html Here is a link to an implementation (I haven't used it myself): http://docs.hazelcast.org/docs/3.3/javadoc/com/hazelcast/client/spi/impl/AwsAddressTranslator.html