Apache Spark and gRPC

问题:

I'm writing an application that uses Apache Spark. For communicating with a client, I would like to use gRPC.

In my Gradle build file, I use

dependencies {
  compile('org.apache.spark:spark-core_2.11:1.5.2')
  compile 'org.apache.spark:spark-sql_2.11:1.5.2'
  compile 'io.grpc:grpc-all:0.13.1'
  ...
}

When leaving out gRPC, everything works fine. However, when gRPC is used, I can create the build, but not execute it, as various versions of netty are used by the packages. Spark seems to use netty-all, which contains the same methods (but with potentially different signatures) than what gRPC uses.

I tried shadowing (using com.github.johnrengelman.shadow) , but somehow it still does not work. How can I approach this problem?

回答1:

The general solution to this sort of thing is shading with relocation. See the answer to a similar problem with protobuf dependencies: https://groups.google.com/forum/#!topic/grpc-io/ABwMhW9bU34

回答2:

I think the problem is that spark uses netty 4.0.x and gRPC 4.1.0 .