So many tomcat http-worker threads are blocked esp

问题:

Environment: Apache-tomcat-7.0.53, CentOS release 6.6 (Final), Java 1.8, Mongo 2.6

Design of our App

We have most of the calls handled with normal servlets(traditional synchronous ones). We do have a 2 asynchronous servlets(servlet 3.0) which delegate their work to ThreadPoolExecutors(threads named http-async-workers).

Problem: We have a database heavy application, but it was performing quite well until 3500ccu. But when we ramped up the CCU to 7000CCU, to measure the scalability factor, we have a lot waiting at the following method org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.register

This is the profiler snapshot:

This is the blocked threads view:

This is the how the threads look like(most of the them are blocked)

Query

Why is org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.register blocked than the database calls? Is it just a symptom of the db issues or the problem itself.

Any help is much appreciated.