java.lang.NoSuchMethodError: akka.actor.ActorCell.

问题:

I am trying to setup a simple akka-http 2.4.2 project to test it out, but I am failing to do so.

My built.sbt:

import NativePackagerHelper._

lazy val akkaVersion = "2.4.2"

lazy val root = (project in file(".")).
settings(
name := "akkTest",
version := "0.1",
scalaVersion := "2.11.7")

libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-http-spray-json-experimental" % akkaVersion
)

enablePlugins(JavaServerAppPackaging)

my code snippet in Main.scala

import akka.http.scaladsl.Http
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.actor.ActorSystem


object Main extends App {

implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
implicit val ec = system.dispatcher

val serverSource =
    Http().bind(interface = "localhost", port = 8080)
val bindingFuture =
    serverSource.to(Sink.foreach { connection => // foreach materializes the source
    println("Accepted new connection from " + connection.remoteAddress)
    }).run()

}

The error on execution throws:

Uncaught error from thread [default-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[default]
java.lang.NoSuchMethodError: akka.actor.ActorCell.addFunctionRef(Lscala/Function2;)Lakka/actor/FunctionRef;
        at akka.stream.stage.GraphStageLogic$StageActor.<init>(GraphStage.scala:143)
        at akka.stream.stage.GraphStageLogic.getStageActor(GraphStage.scala:904)
        at akka.stream.impl.io.ConnectionSourceStage$$anon$1.preStart(TcpStages.scala:56)
        at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:468)
        at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:363)
        at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:502)
        at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:539)
        at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:493)
        at akka.actor.ActorCell.create(ActorCell.scala:580)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
        at akka.dispatch.Mailbox.run(Mailbox.scala:220)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This must be something on my environment but I dont know how to track the issue. I am using jdk 1.8u71

[info] Done updating.
[info] Including from cache: ssl-config-akka_2.11-0.1.3.jar
[info] Including from cache: reactive-streams-1.0.0.jar
[info] Including from cache: akka-http-spray-json-experimental_2.11-2.4.2.jar
[info] Including from cache: config-1.3.0.jar
[info] Including from cache: spray-json_2.11-1.3.2.jar
[info] Including from cache: ssl-config-core_2.11-0.1.3.jar
[info] Including from cache: scala-parser-combinators_2.11-1.0.4.jar
[info] Including from cache: scala-java8-compat_2.11-0.7.0.jar
[info] Including from cache: akka-parsing_2.11-2.4.2.jar
[info] Including from cache: akka-http-experimental_2.11-2.4.2.jar
[info] Including from cache: akka-actor_2.11-2.4.2.jar
[info] Including from cache: akka-http-core_2.11-2.4.2.jar
[info] Including from cache: akka-stream_2.11-2.4.2.jar
[info] Including from cache: scala-library-2.11.7.jar

Take in mind I only point to dependencies of the same akka version

This program works fine when using sbt run but fails when using the assembled jar with my own scala launcher

回答1:

The problem is that Spark uses Akka internally. As long as you run your Spark Job as a self contained application (e.g. sbt run), this is not a problem, since your own version of Akka will be used. Things change however as soon as you submit your application to a cluster with spark-submit. The Spark classloader will then pick the internal version of Akka over the Akka implementation bundled in your sparkJob.jar. The NoSuchMethodError from above therefore comes from akka-stream_2.11-2.4.2 calling into akka-actor_2.11-2.3.x.jar which is used in Spark, instead of akka-actor_2.11-2.4.2.jar which is bundled in your job. The method addFunctionRef is in fact a very recent addition and is not present in earlier versions of Akka. You can verify this by setting a breakpoint at the place where the exception occurs (or use an exception breakpoint). With the application suspend at the problematic location in GraphStage, evaluate

materializer.supervsor.getClass().getResource("ActorCell.class")

This will print out the location of the class file ActorCell was loaded from.

To make sure that you are isolated from the Akka version that Spark uses internally, you can use the --driver-class-path option of spark-submit, like

spark-submit --class MyJob \
  --driver-class-path akka-actor_2.11-2.4.2.jar \
  --master spark://master:7077 \
  sparkJob.jar

If you do this, I also recommend setting akka-actor to "test, provided" in your build.sbt, so that you don't also include akka-actor in sparkJob.jar.

回答2:

Scala 2.11 bundles an old version of akka-actor. Scala puts their libraries to take precedence over user made libraries. Spark might bundle the same, although inside it's assembly Jar.

In scala, deleting akka jar from lib folder is enough. In Spark, utilizing driver-class-path resolves the issue, or utilize SQLContext configuration to make user libraries take precedence

Unanswered question is why this does not happen in all setup, as this did not happen at home (Maybe OS repository does not install scala with akka-actor library)

https://github.com/sbt/sbt-assembly/issues/204

回答3:

When this thing happened to me, I resolved it by eliminating a transitive dependency on an older akka-actor version. Even though the dependencies look obvious to you, i would suggest looking closer at your dependency tree to see if an earlier version of akka-actor is being included.

回答4:

For those if it happens even with sbt run. The solution for me was to add akka-stream dependency in build.sbt file, e.g:

  "com.typesafe.akka" %% "akka-stream" % "2.5.3",
  "com.typesafe.akka" %% "akka-stream-testkit" % "2.5.3" % Test

回答5:

I had similar issue while using akka-http-experimental-2.4.11.2 with akka-actor-2.5.6.

dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.11'
    compile group: 'com.typesafe.akka', name: 'akka-actor_2.11', version: '2.5.6'

    compile group: 'com.typesafe.akka', name: 'akka-http-experimental_2.11', version: '2.4.11.2'
}

And the problem was, akka-http-experimental-2.4.11.2 itself uses com.typesafe.akka:akka-actor:2.4.11.2. which was overridden by 2.5.6.

./gradlew dependencies

compileClasspath - Compile classpath for source set 'main'.
+--- org.scala-lang:scala-library:2.11.11
+--- com.typesafe.akka:akka-actor_2.11:2.5.6
|    +--- org.scala-lang:scala-library:2.11.11
|    +--- com.typesafe:config:1.3.1
|    \--- org.scala-lang.modules:scala-java8-compat_2.11:0.7.0
|         \--- org.scala-lang:scala-library:2.11.7 -> 2.11.11
+--- com.typesafe.akka:akka-http-experimental_2.11:2.4.11.2
|    +--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
|    \--- com.typesafe.akka:akka-http-core_2.11:2.4.11.2
|         +--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
|         +--- com.typesafe.akka:akka-stream_2.11:2.4.11.2
|         |    +--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
|         |    +--- com.typesafe.akka:akka-actor_2.11:2.4.11.2 -> 2.5.6 (*)  <---- Ting
|         |    +--- com.typesafe:ssl-config-akka_2.11:0.2.1
|         |    |    +--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
|         |    |    \--- com.typesafe:ssl-config-core_2.11:0.2.1
|         |    |         +--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
|         |    |         +--- com.typesafe:config:1.2.0 -> 1.3.1
|         |    |         \--- org.scala-lang.modules:scala-parser-combinators_2.11:1.0.4
|         |    |              \--- org.scala-lang:scala-library:2.11.6 -> 2.11.11
|         |    \--- org.reactivestreams:reactive-streams:1.0.0
|         \--- com.typesafe.akka:akka-parsing_2.11:2.4.11.2
|              \--- org.scala-lang:scala-library:2.11.8 -> 2.11.11
\--- org.codehaus.groovy:groovy-all:2.4.12

(*) - dependencies omitted (listed previously)

1) Once I lowered the akka-actor version to 2.4.11, it works fine.

dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.11'
    compile group: 'com.typesafe.akka', name: 'akka-actor_2.11', version: '2.4.11'
    compile group: 'com.typesafe.akka', name: 'akka-http-experimental_2.11', version: '2.4.11.2'
}

2) The other way is to exclude the unwanted akka_actor and add explicitly the wanted one.

dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.11'
    compile group: 'com.typesafe.akka', name: 'akka-actor_2.11', version: '2.5.6'
    compile group: 'com.typesafe.akka', name: 'akka-http-experimental_2.11', version: '2.4.11.2' {
       exclude group: "com.typesafe.akka", module: "akka-actor_2.11"         // <- exlude actor 2.4.11.2
    }
}