Google Cloud BigQuery Import not working in app en

Question

I have used the following code to build an app engine project to move data from google cloud bucket into the bigquery table

import argparse
import time
import uuid

from google.cloud import bigquery


def load_data_from_gcs(dataset_name, table_name, source):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)
    job_name = str(uuid.uuid4())

    job = bigquery_client.load_table_from_storage(
        job_name, table, source)

    job.begin()

    wait_for_job(job)

    print('Loaded {} rows into {}:{}.'.format(
        job.output_rows, dataset_name, table_name))


def wait_for_job(job):
    while True:
        job.reload()
        if job.state == 'DONE':
            if job.error_result:
                raise RuntimeError(job.error_result)
            return
        time.sleep(1)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('dataset_name')
    parser.add_argument('table_name')
    parser.add_argument(
        'source', help='The Google Cloud Storage object to load. Must be in '
        'the format gs://bucket_name/object_name')

    args = parser.parse_args()

    load_data_from_gcs(
        args.dataset_name,
        args.table_name,
        args.source)

I have also altered the default app.yaml file as the above file and deleted the webapp2 library entry and my app.yaml file looks like this

application: gcstobq
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon\.ico
  static_files: favicon.ico
  upload: favicon\.ico

- url: .*
  script: main.app

As I am new to python and app engine I dont know if I need to include the libraries defines in main.py file into the app.yaml and if i need to run this app using the command line tool.

Please let me know if I am missing something here?

Answer 1

Google Cloud uses the new Python namespace format (if you look at the source you'll notice that there's no __init__.py in the directory structure). This was changed in Python 3.3 with PEP-420

Fortunately in Python 2.7 you can fix this easily by avoiding implicit imports. Just add this to the very top of your file (before any other imports) to get the Python 3 behavior:

from __future__ import absolute_import