Slow topology - Uneven load on Executers in a Stor

问题:

Structure to the problem :

4 node cluster
Bolt 2 - parallelism factor is 8, so 2 executers per worker.

I have notice lag overtime in the storm topology due to high latency in one of the executers.

If you'll notice in the pic, only 1 executer is having slightly more load and significantly high latency. Spout is working great! Any pointer to the possible issues appreciated.

Noting unusual observer in worker/supervisor logs.

The worker JVM has high has huge memory so that is not a issue.

PS: cpu utilisation at the time of issue was >100% cumulative whereas on other nodes it was ~15%. This is already narrowed that it is not a machine specific issue as this behaviour switches from one Executer to other in a cycle of few hours.