Unable to Write to bigquery - Permission denied: A

问题:

I have been using apache beam python sdk using google cloud dataflow service for quite some time now.

I was setting dataflow up for a new project.

The dataflow pipeline

1. Reads data from google datastore
2. Processes it
3. Writes to Google Big-Query.

I have similar pipelines running on other projects which are running perfectly fine.

Today, When I started a dataflow job, the pipeline started, read data from datastore, processed it and when it was about to write it to bigquery, It resulted in

apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: 
Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: S04:read from datastore/GroupByKey/Read+read 
from datastore/GroupByKey/GroupByWindow+read from datastore/Values+read 
from datastore/Flatten+read from datastore/Read+convert to table 
rows+write to bq/NativeWrite failed., BigQuery import job 
"dataflow_job_8287310405217525944" failed., BigQuery creation of import 
job for table "TableABC" in dataset "DatasetABC" in project "devel- 
project-abc" failed., BigQuery execution failed., Error:
Message: Access Denied: Dataset devel-project-abc:DatasetABC: The user 
[e-mail] does not 
have bigquery.tables.create permission for dataset devel-project- 
abc:DatasetABC: HTTP Code: 403

I made sure all the required API are enabled. According to me the service account has the necessary permission.

My question is Where this might be going wrong?

Update

From what I remember on previous projects (3 different projects to be precise) I didn't give the dataflow service agent any specific permission. The compute engine service agent had permissions like dataflow admin, editor, dataflow viewer. Hence before proceeding with giving the service agent permissions related to bigquery, i would like to know why the environment is behaving differently than the previous projects.

Is there any permission/policy changes/updates that went live in last few months resulting in requirement of bigquery writer permission?

回答1:

Please make sure your service account ('[e-mail]') has 'roles/bigquery.dataEditor' role in 'devel-project-abc:DatasetABC'. Also make sure 'BigQuery Data Editor' role is enabled for your project.

GCP IAM is where you can check those.

回答2:

You can find the capabilities for each role for BigQuery here. If your previous projects were using primitive IAM roles, then you might need to set correctly. IAM Release Notes page is provided here which provides additional information on the updates done to the system.

回答3:

Your question is not clear!? If you are asking Why data flow can't write to BigQuery? then the answer should be related to the permissions you gave to the service-account you are using. Check Michael Moursalimov answer...

But, If you are asking What the deferent between your old project and the new one? Then I can't answer, ask GCP support, or just spend more time comparing your settings for both projects.