I'm running GCP Dataflow job when I'm reading data from BigQuery as a query result. I'm using google-cloud-dataflow-java-sdk-all version 1.9.0. The code fragment that sets up the pipeline looks like this:
PCollection<TableRow> myRows = pipeline.apply(BigQueryIO.Read
.fromQuery(query)
.usingStandardSql()
.withoutResultFlattening()
.named("Input " + tableId)
);
The query is quite complex what results in error message:
Query exceeded resource limits for tier 1. Tier 8 or higher required., error: Query exceeded resource limits for tier 1. Tier 8 or higher required.
I'd like to set maximumBillingTier
as it is done in Web UI or in bq script. I can't find any way to do so except for setting default for the entire project which is unfortunately not an option.
I tried to set it through these without success:
- DataflowPipelineOptions - neither this nor any interface it extends seems to have that setting
- BigQueryIO.Read.Bound - I would expect it to be there just next to
usingStandardSql
and others similar but obviously it is not there - JobConfigurationQuery - this class has all cool settings but it seems it is not used at all when setting up a pipeline
Is there any way to pass this setting from within Dataflow job?