Does Dataflow jobs hit any Bigquery quotas and lim

Question

I have around 1500 jobs to be implemented using Dataflow. Those jobs will be scheduled on daily basis. We may get to use huge number of DML statements using Bigquery Client library within our jobs. Listing down my concerns regarding Bigquery quotas and limits.

Reference: https://cloud.google.com/bigquery/quotas

Please confirm that do we need to take the daily usage limits of Bigquery into consideration in any of the below mentioned scenarios.

1. If we implement data inserts using BigqueryIO.write()
2. If we use DML statements (Update/Delete) using Bigquery Client Library within the Dataflow job

Please suggest.

Answer 1

You absolutely do need to take BigQuery quotas and limits into consideration - even when hooking into it from Dataflow.

Dataflow is just calling the BigQuery API on your behalf. Therefore, all quotas and limits still apply as if you were calling it directly yourself.