Slow transactions in NewRelic having Play Framewor

could someone help me to read NewRelic Summary and Trace details. Following screenshots have trace for a single transaction, which do not create any query to the database. It is just a simple query with few lines of Scala template code, which renders HTML page and returns it to the client. This is just a single transaction that is currently running in production. Production has plenty of more complex transaction running which do lots of external calls to Mongo, Maria, Queue, etc.

Does the trace reveal anything about where bottleneck could be? Are we for example running out of Threads or Workers. As I told most of the transactions do lots of web external calls, which might reserve single Thread for quite long time. How one can actually study if Threads or Workers are running out in Play application? We are using 2.1.4.

What actually happens in following calls?

Promise.apply 21.406ms

Async Wait 21.406ms

Actor.tell 48.366ms

PlayDefaultUpstreamHandler 6.292ms

Edit:

What is the purpose of following calls? Those have super high average call times.

scala.concurrent.impl.CallbackRunnable.run()
scala.concurrent.impl.Future$PromiseCompletingRunnable.run()
org.jboss.netty.handler.codec.http.HttpRequestDecoder.unfoldAndFireMessageReceived()

Edit:

play {
  akka {
    event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
    loglevel = WARNING
    actor {
      default-dispatcher = {
        fork-join-executor {
          parallelism-min = 350
          parallelism-max = 350
        }
      }
      exports = {
        fork-join-executor {
          parallelism-min = 10
          parallelism-max = 10
        }
      }
    }
  }
}

I'm not sure if this will help you 1 year later but I think the performance problems you were hitting are not related to Play, Akka or Netty.

The problem will be in your code business logic or in the database access. The big times that you see for PromiseCompletingRunnable and unfoldAndFireMessageReceived are misleading. This times are reported by newrelic in a wrong and misleading way. Please read this post:

Extremely slow play framework 2.3 request handling code

I faced a similar problem, and mine was in the database but newrelic reported big times in netty.

I hope this helps you even now.