I am trying to create a cluster in Dataproc using google-cloud-python library, however, when setting region = 'us-central1' I get below exception:
google.api_core.exceptions.InvalidArgument: 400 Region 'us-central1' is invalid.
Please see https://cloud.google.com/dataproc/docs/concepts/regional-endpoints
for additional information on regional endpoints
Code (based on example):
#!/usr/bin/python
from google.cloud import dataproc_v1
client = dataproc_v1.ClusterControllerClient()
project_id = 'my-project'
region = 'us-central1'
cluster = {...}
response = client.create_cluster(project_id, region, cluster)
Dataproc uses region field for routing REST requests, however, the field is not used in gRPC clients (hence the error).
Only the global multiregion can be accessed through the default endpoint. To use a regional endpoint such as us-central1, you have to configure the endpoint to address on the client's transport.
The Dataproc regional endpoints follow this pattern: <region>-dataproc.googleapis.com:443. The region field should be set to the same value as the region in the endpoint.
Example:
#!/usr/bin/python
from google.cloud import dataproc_v1
from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport
transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport(
address='us-central1-dataproc.googleapis.com:443')
client = dataproc_v1.ClusterControllerClient(transport)
project_id = 'my-project'
region = 'us-central1'
cluster = {...}
response = client.create_cluster(project_id, region, cluster)
Similarly using the google-cloud-java client:
ClusterControllerSettings settings =
ClusterControllerSettings.newBuilder()
.setEndpoint("us-central1-dataproc.googleapis.com:443")
.build();
try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settings)) {
String projectId = "my-project";
String region = "us-central1";
Cluster cluster = Cluster.newBuilder().build();
Cluster response =
clusterControllerClient.createClusterAsync(projectId, region, cluster).get();
}
