Spark wordcount assertion failed: unsafe symbol Un

I have installed HDFS, Yarn and Spark using Hortonworks Ambari. I've written simple programs to read/read to HDFS, Map-Reduce wordcount, all worked fine.

I then tried to test Spark. I copied the word count program from official Spark example:

public final class JavaWordCount {
  private static final Pattern SPACE = Pattern.compile(" ");

  public static void main(String[] args) throws Exception {

    if (args.length < 1) {
      System.err.println("Usage: JavaWordCount <file>");
      System.exit(1);
    }

    SparkSession spark = SparkSession
      .builder()
      .appName("JavaWordCount")
      .getOrCreate();

    JavaRDD<String> lines = spark.read().textFile(args[0]).javaRDD();

    JavaRDD<String> words = lines.flatMap(s -> Arrays.asList(SPACE.split(s)).iterator());

    JavaPairRDD<String, Integer> ones = words.mapToPair(s -> new Tuple2<>(s, 1));

    JavaPairRDD<String, Integer> counts = ones.reduceByKey((i1, i2) -> i1 + i2);

    List<Tuple2<String, Integer>> output = counts.collect();
    for (Tuple2<?,?> tuple : output) {
      System.out.println(tuple._1() + ": " + tuple._2());
    }
    spark.stop();
  }
}

I put an input.txt file with content "Hello World" in hdfs://server:8020/user/root/input.txt.

I then run this program using the following command,

java -cp  "all necessary jars provided by Ambari, I believe the version should match" JavaWordCount hdfs://server:8020/user/root/input.txt

There is then very strange exception that even I searched on google I could not find any similar match.

The exception occurs at line 27 which is

JavaRDD<String> lines = spark.read().textFile(args[0]).javaRDD();

And the exception is quite strange that I cannot find any exact match on search engine.

Is there any hints for me to check and look into to make it work? Thanks. The exception is very long and I list them at last.

java.lang.AssertionError: assertion failed: unsafe symbol Unstable (child of package InterfaceStability) in runtime reflection universe
        at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:184)
        at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3009)
        at scala.reflect.internal.Symbols$ClassSymbol.<init>(Symbols.scala:3201)
        at scala.reflect.internal.Symbols$StubClassSymbol.<init>(Symbols.scala:3496)
        at scala.reflect.internal.Symbols$Symbol.newStubSymbol(Symbols.scala:498)
        at scala.reflect.internal.pickling.UnPickler$Scan.readExtSymbol$1(UnPickler.scala:258)
        at scala.reflect.internal.pickling.UnPickler$Scan.readSymbol(UnPickler.scala:284)
        at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolRef(UnPickler.scala:649)
        at scala.reflect.internal.pickling.UnPickler$Scan.readType(UnPickler.scala:417)
        at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
        at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
        at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:179)
        at scala.reflect.internal.pickling.UnPickler$Scan.readTypeRef(UnPickler.scala:658)
        at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotationInfo(UnPickler.scala:492)
        at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolAnnotation(UnPickler.scala:515)
        at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:97)
        at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:38)
        at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
        at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
        at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
        at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
        at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
        at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
        at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
        at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
        at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
        at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
        at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
        at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
        at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
        at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
        at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
        at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
        at scala.reflect.internal.Types$Type.member(Types.scala:600)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
        at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
        at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:161)
        at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1$$typecreator46$1.apply(ScalaReflection.scala:654)
        at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
        at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
        at org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:839)
        at org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:39)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply$mcZ$sp(ScalaReflection.scala:654)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply(ScalaReflection.scala:653)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply(ScalaReflection.scala:653)
        at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
        at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:824)
        at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:39)
        at org.apache.spark.sql.catalyst.ScalaReflection$.optionOfProductType(ScalaReflection.scala:652)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:51)
        at org.apache.spark.sql.Encoders$.STRING(Encoders.scala:96)
        at org.apache.spark.sql.SQLImplicits.newStringEncoder(SQLImplicits.scala:73)
        at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:732)
        at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:702)
        at JavaWordCount .main(JavaWordCount .java:27)
Exception in thread "main" java.lang.RuntimeException: error reading Scala signature of org.apache.spark.sql.package: assertion failed: unsafe symbol Unstable (child of package InterfaceStability) in runtime reflection universe
        at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:46)
        at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
        at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
        at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
        at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
        at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
        at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
        at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
        at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
        at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
        at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
        at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
        at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
        at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
        at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
        at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
        at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
        at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
        at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
        at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
        at scala.reflect.internal.Types$Type.member(Types.scala:600)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
        at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
        at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
        at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:161)
        at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1$$typecreator46$1.apply(ScalaReflection.scala:654)
        at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
        at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
        at org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:839)
        at org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:39)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply$mcZ$sp(ScalaReflection.scala:654)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply(ScalaReflection.scala:653)
        at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$optionOfProductType$1.apply(ScalaReflection.scala:653)
        at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
        at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:824)
        at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:39)
        at org.apache.spark.sql.catalyst.ScalaReflection$.optionOfProductType(ScalaReflection.scala:652)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:51)
        at org.apache.spark.sql.Encoders$.STRING(Encoders.scala:96)
        at org.apache.spark.sql.SQLImplicits.newStringEncoder(SQLImplicits.scala:73)
        at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:732)
        at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:702)
        at JavaWordCount .main(JavaWordCount .java:27)

标签： java apache-spark hadoop ambari

1条回答

该账号已被封号

2楼-- · 2019-08-25 03:04

Try to submit spark job with spark-submit script, you cannot use java command directly, although spark-submit also using java command.

0人赞添加讨论(0) 举报

Spark wordcount assertion failed: unsafe symbol Un

采纳回答

编辑标签

举报内容

检举类型

检举原因

检举说明(必填)

打开微信“扫一扫”，打开网页后点击屏幕右上角分享按钮

付费偷看金额在0.1-10元之间