Exceptions in Hbase Region Server

Question

I continuously see these logs getting printed in hbase region server logs. 10.0.5.133 is the machine where hbase java client is present.

It keeps on disconnecting and connecting. Any reason why this is happening ? Any mismatch in client library /server version ? Any other reason ? I tried to google this but could not found any helpful link.

2015-12-25 20:24:03,210 DEBUG [RpcServer.reader=9,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54538 because read count=-1. Number of active connections: 30
2015-12-25 20:24:03,213 DEBUG [RpcServer.reader=2,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54564 because read count=-1. Number of active connections: 29
2015-12-25 20:24:08,350 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:54642; # active connections: 26
2015-12-25 20:24:18,358 DEBUG [RpcServer.reader=8,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54642 because read count=-1. Number of active connections: 29
2015-12-25 20:24:30,543 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:54799; # active connections: 28
2015-12-25 20:24:40,547 DEBUG [RpcServer.reader=0,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54799 because read count=-1. Number of active connections: 29
2015-12-25 20:24:41,870 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:54864; # active connections: 29
2015-12-25 20:24:51,877 DEBUG [RpcServer.reader=8,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54864 because read count=-1. Number of active connections: 28
2015-12-25 20:25:00,634 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:54982; # active connections: 27
2015-12-25 20:25:10,645 DEBUG [RpcServer.reader=6,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:54982 because read count=-1. Number of active connections: 30
2015-12-25 20:26:30,169 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55511; # active connections: 26
2015-12-25 20:26:40,176 DEBUG [RpcServer.reader=3,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55511 because read count=-1. Number of active connections: 32
2015-12-25 20:26:41,117 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55594; # active connections: 30
2015-12-25 20:26:44,607 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55623; # active connections: 31
2015-12-25 20:26:51,129 DEBUG [RpcServer.reader=7,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55594 because read count=-1. Number of active connections: 26
2015-12-25 20:26:54,614 DEBUG [RpcServer.reader=2,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55623 because read count=-1. Number of active connections: 23
2015-12-25 20:27:01,926 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55731; # active connections: 31
2015-12-25 20:27:11,938 DEBUG [RpcServer.reader=5,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55731 because read count=-1. Number of active connections: 29
2015-12-25 20:27:14,711 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55776; # active connections: 28
2015-12-25 20:27:24,719 DEBUG [RpcServer.reader=3,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55776 because read count=-1. Number of active connections: 28
2015-12-25 20:27:36,791 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:55938; # active connections: 24
2015-12-25 20:27:45,008 DEBUG [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: connection from 10.0.5.133:56013; # active connections: 27
2015-12-25 20:27:46,799 DEBUG [RpcServer.reader=4,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: DISCONNECTING client 10.0.5.133:55938 because read count=-1. Number of active connections: 29

Answer 1

Looks like the client connection load is way more than default max client connection limit of 30. You might want to consider bumping up that configuration.

Here's more information: http://archive.cloudera.com/cdh/3/hbase-0.90.1-cdh3u0/configuration.html

hbase.zookeeper.property.maxClientCnxns

Property from ZooKeeper's config zoo.cfg. Limit on number of concurrent connections (at the socket level) that a single client, identified by IP address, may make to a single member of the ZooKeeper ensemble. Set high to avoid zk connection issues running standalone and pseudo-distributed.

Default: 30