Play framework [2.5.0 java] - Blocked netty-event-

问题:

We have just upgraded from Play framework 2.4.3 to 2.5.0 (java). However, after upgrading, our tests start getting timeout after a couple of minutes. Before the upgrade they ran for an hour without errors.

It looks like some threads are getting blocked, and the system simply stops responding.

I am running a smaller version of the load test locally on my machine, with Yourkit java profiler. Initially, there are 16 netty-event-loop threads started. After about a minute, I can see that they have started blocking:

When they block I start getting timeouts in the load test. When I turn off the test, these threads seem to recover:

I am hoping someone here can help us determine what is causing this. We have not modified our code at all apart from the changes needed to upgrade to Play 2.5.

Here is the akka thread pool config we're using in application.conf:

akka {
  fork-join-executor {
    # The parallelism factor is used to determine thread pool size using the
    # following formula: ceil(available processors * factor). Resulting size
    # is then bounded by the parallelism-min and parallelism-max values.
    parallelism-factor = 3.0

    # Min number of threads to cap factor-based parallelism number to
    parallelism-min = 8

    # Max number of threads to cap factor-based parallelism number to
    parallelism-max = 64

    # Setting to "FIFO" to use queue like peeking mode which "poll" or "LIFO" to use stack
    # like peeking mode which "pop".
    task-peeking-mode = "FIFO"
  }
}

The profiler shows the following info about the blocked threads:

Can anyone provide some insight into what we might be doing wrong? Thanks for your help.

回答1:

This issue seems to be resolved for us. We were using Deadbolt-java 2.5.0-SNAPSHOT for authorization in templates and controllers. We were seeing some timeout messages in our logs related to Deadbolt.

So we completely removed Deadbolt from our project, and now the load tests run faster than ever.