Unexpected exception during request

问题:

I'm using apache cassandra 2.2.4. I have a 4 (four) node cluster with Replication Factor 3 in DC1 and Replication Factor 1 in DC2, where DC1 contains 3 (three) nodes and DC2 contains 1 (one) node. There were some more nodes before in this cluster, but for some reason I removed them and didn't altered the replication scenario. [Please consider that the following IP's are not original]

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  21.12.19.91    4.08 GB    256          ?       a45bb676-1ddd-4b22-933b-58653cea680f  RAC1
UN  21.12.19.92    3.92 GB    256          ?       a7735fca-8671-4a20-a759-4a2681aed37e  RAC1
UN  21.12.19.93    4.47 GB    256          ?       d98f3cad-881a-41c8-89c7-170c63c3d236  RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  21.12.19.99    3.84 GB    256          ?       ccd9ca97-f97a-4473-9a65-49b12a1b60ba  RAC1

Cluster was working fine, but now-a-days I'm having an issue as INFO. I was trying to analyze the issue, but couldn't make it yet. Is there anyone, who is familiar with following scenario?

INFO  [SharedPool-Worker-2] 2017-02-26 06:56:48,520 Message.java:605 - Unexpected exception during request; channel = [id: 0x637a702c, /18.12.10.17:60926 :> /21.12.19.91:9042]
java.io.IOException: Error while read(...): Connection reset by peer
    at io.netty.channel.epoll.Native.readAddress(Native Method) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

回答1:

Please make sure that your Firewall is not dropping TCP connections which are in use. Tcp keep alive on all nodes must be less than the firewall setting. Please refer to https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html for details on TCP settings. This helped me resolve the issue.