Debugging akka performance

问题:

I've working on an Akka application where each actor invokes a web service. When a performance test is run the actors start to respond very slowly after approx 6 minutes. I've created a unit test to test the performance and not seeing any issues. I realize this is a generic question but are there general areas of the application I can investigate to discover where the issue may lie ?

One area I'm considering is routing : http://doc.akka.io/docs/akka/snapshot/scala/routing.html

回答1:

First of all check general condition of the application using standard tools jvisualvm, jconsole.

If all the basics parameters (cpu, memory, threads state) are correct, then you should try monitoring particular actors using kamon.io and garfana/statsd/graphite stack docker-config

You can follow get-started manual and apply following configuration in your application.conf

kamon
{
  metrics
  {
    actor {
      filters = [
        {
          actor {
            includes = ["*"]
            excludes = ["system/*", "user/IO-*"]
          }
        },
        {
          router {
            includes = ["*"]
            excludes = ["system/*", "user/IO-*"]
          }
        },
        {
          trace {
            includes = ["*"]
            excludes = []
          }
        },
        {
          dispatcher {
            includes = ["default-dispatcher"]
            excludes = []
          }
        }
      ]
    }
  }
  statsd {
    hostname = "10.0.1.4"
    port = 8125
    flush-interval = 1 second
    max-packet-size = 1024 bytes

    includes
    {
      actor = ["*"]
      trace = ["*"]
      dispatcher = ["*"]
    }
    simple-metric-key-generator {

      application = "test.akka"
    }
  }
}

回答2:

Given that you just invoking remote web services it's unlikely that your system is CPU bound. Here are possible things you should look at (using a profiling tool, I highly recommend YourKit. If don't want to pay for the license you can use VisualVM). Since you have not posted any sample code I cannot say if any of the below is true in your case.

1. There is blocking code in the receive method of your actors. This means you will quickly run out of threads if takes way too long for the web service to return and you are blocking on them to return.

2. You have not given enough memory to your actor-system.

3. There is a memory leak. For example, are you stopping/killing the actors after they are done with their web service call or are you reusing the actors

Finally as @hicolour mentioned kamon.io is a great tool to find performance bottlenecks.