Error HBASE-ZOOKEEPER : Too many connections

Question

I am using Hbase-Hadoop combination for my application along with Data Nucleus as the ORM.

When I am trying to access hbase via several threads at a single time. It throws exceptions as :

Exception in thread "Thread-26" javax.jdo.JDODataStoreException

org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

If required I can provide the complete stack trace.(Since the complete stack trace makes thing messy out here).

Kindly help me out with some leads on how to handle this kind of situation. Is there any configuration I need to do to increase the connection pool?

Answer 1

I don't suggest to blindly dump up the max connection, this config is not max connections to ZK, but max concurrent connections per client to ZK, according to the ZK document:

maxClientCnxns (No Java system property)

Limits the number of concurrent connections (at the socket level) that a single client, identified by IP address, may make to a single member of the ZooKeeper ensemble. This is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. Setting this to 0 or omitting it entirely removes the limit on concurrent connections.

therefore, you must have more than 30 concurrent connections to ZK from the same IP address. in this case, use netstat to debug what that IP is, and inspect what could init those many connections at the same time.

Answer 2

Zookeeper servers have an active connections limit, which by default is 30. You need to increase this limit by setting maxClientCnxns property accordingly in your zookeeper config file, zoo.cfg.

For 100 connections:

maxClientCnxns=100

To tell zookeeper to impose no limit to the number of connections:

maxClientCnxns=0

Answer 3

For the above issue you need to restart the master service of HBase. The command for the same is

sudo /etc/init.d/hadoop-hbase-master start

this needs to be done before entering to HBase shell.