importing data from firestore into bigQuery

Question

I have upgraded my account to blaze which is one of the prerequisite. I have tried to followed the FAQ Link BigQuery to Firebase. still not able to see any of the data from firestore or firebase-realtimeDatabase into bigQuery.

I see option in bigQuery to create Dataset. however after creating dataset it allows me to upload data from [file], [cloud storage], [bigQuery],[google Drive] . but not firestore database.

Plesae help.

Answer 1

Guys there is an example for importing data into bigquery

https://github.com/firebase/functions-samples/tree/master/bigquery-import

luck..

Answer 2

In case anyone need it. I ended up automating scripts because current export data option only allows stream data and preserve for 30 days.

export data

gcloud beta firestore export --collection-ids=users gs://mybucket/users

load backup into bq

bq load --source_format=DATASTORE_BACKUP mydataset.users gs://gs://mybucket/users/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata

Answer 3

There is no built-in support to import data from the Firebase Realtime Database or Cloud Firestore into BigQuery.

For now, if you want to import data, you'll have to write code to do so.

Answer 4

Firestore now allows to import / export data. The data is exported to a Cloud Storage bucket and from there it can be imported into Big Query. Here are some links that might be helpful:

• https://firebase.google.com/docs/firestore/manage-data/export-import
• https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore (Firestore uses the same format as Datastore for imports / exports)

** Edit: Docs for BigQuery imports from Firestore are now available https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore

Answer 5

I made an NPM package that lets you create a BigQuery dataset and tables with autogenerated schemas based on your Firestore data, and then copy and convert chosen Firestore collections.

https://www.npmjs.com/package/firestore-to-bigquery-export