Dataflow fails with java.lang.NoSuchMethodError: i

问题:

I'm trying to get a Dataflow job to run on Google Cloud. It always fails with:

java.lang.NoSuchMethodError: io.grpc.protobuf.ProtoUtils.marshaller(Lcom/google/protobuf/Message;)Lio/grpc/MethodDescriptor$Marshaller

It's a maven project, here are my dependencies:

<dependencies>
  <dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>1.8.0</version>
  </dependency>   
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
    <version>0.4.0</version>
  </dependency>    
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.0.0</version>
  </dependency>       
</dependencies>

I've tried a bunch of different dependency versions. What am I missing?

It has the same result whether I run via exec:java or via a shade jar.

Full stack trace:

(e8dbd0c1b8b8a22): java.lang.NoSuchMethodError: io.grpc.protobuf.ProtoUtils.marshaller(Lcom/google/protobuf/Message;)Lio/grpc/MethodDescriptor$Marshaller; at com.google.iam.v1.IAMPolicyGrpc.(IAMPolicyGrpc.java:56) at com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.(PublisherSettings.java:487) at com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.createDefault(PublisherSettings.java:508) at com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.access$000(PublisherSettings.java:402) at com.google.cloud.pubsub.spi.v1.PublisherSettings.defaultBuilder(PublisherSettings.java:224) at com.google.cloud.pubsub.spi.DefaultPubSubRpc.(DefaultPubSubRpc.java:138) at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:60) at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:54) at com.google.cloud.ServiceOptions.rpc(ServiceOptions.java:399) at com.google.cloud.pubsub.PubSubImpl.(PubSubImpl.java:115) at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:43) at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:38) at com.google.cloud.ServiceOptions.service(ServiceOptions.java:391) at com.google.lindsaysmith.titan.DataflowBulkLoadNodes$SendPubSub.sendPubsubMessage(DataflowBulkLoadNodes.java:41) at com.google.lindsaysmith.titan.DataflowBulkLoadNodes$SendPubSub.processElement(DataflowBulkLoadNodes.java:32) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55) at

回答1:

The usual answer in this situation is that you really must use exactly the version of gRPC and Protocol Buffers declared in the dependencies of the Dataflow SDK. This includes all transitive dependencies, so you may have to suppress gRPC or protobuf dependencies of other libraries so they do not intefere.

You can see the versions here (gRPC) and here (protobuf). I'm leaving them out of this answer so it does not get out of date.