Cannot get gcloud to work with Python and Pycharm

Question

I am trying to connect to the Google App Engine Datastore from my local machine. I have spent all day digging in to this without any luck.

I have tried the approach here (as well as alot of other suggestions from SO such as Using gcloud-python in GAE and Unable to run dev_appserver.py with gcloud):

How to access a remote datastore when running dev_appserver.py?

I first installed gcloud based on this description from google: https://cloud.google.com/appengine/docs/python/tools/using-libraries-python-27

According to the description I should add the following to my appengine_config.py:

from google.appengine.ext import vendor
vendor.add('lib')

If I do that I get an error saying ImportError: No module named gcloud

If I then move the code to my main.py it seems to pickup the lib-folder and the modules there. That seems a bit strange to me, since I thought appengine_config was being run first to make sure things were initialised. But now I am getting the following stack trace:

ERROR    2016-09-23 17:22:30,623 cgi.py:122] Traceback (most recent call last):
  File "/Users/thomasd/Documents/github/myapp/main.py", line 10, in <module>
    from gcloud import datastore
  File "/Users/thomasd/Documents/github/myapp/lib/gcloud/__init__.py", line 17, in <module>
    from pkg_resources import get_distribution
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2985, in <module>
    @_call_aside
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2971, in _call_aside
    f(*args, **kwargs)
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 3013, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2544, in activate
    declare_namespace(pkg)
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2118, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2057, in _handle_ns
    loader.load_module(packageName)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py", line 246, in load_module
    mod = imp.load_module(fullname, self.file, self.filename, self.etc)
  File "/Library/Python/2.7/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
  File "/usr/local/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 999, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named google.cloud.logging.client

What am I doing wrong here?

Answer 1

I solved it this way:-

1.) Create a lib folder in your project path.

2.) Install gcloud libraries by running following command into terminal from your project path:-

    pip install -t lib gcloud

3.) Create an appengine_config.py module in your project and add following lines of code:-

    import sys
    import os.path
    sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

4.) After this, you can import like this:-

    from gcloud import datastore

5.) To save data into live google datastore from local:-

    client = datastore.Client("project-id")
    key = client.key('Person')
    entity = datastore.Entity(key=key)
    entity['name'] = ashish
    entity['age'] = 23
    client.put(entity)

It will save an entity named Person having properties name and age. Do not forget to specify your correct project id.

Answer 2

Old question but this may be worth including:

I'm unsure the state of your requirements.txt file but I scrounged mine a bit and noticed setuptools was not included.

pip freeze doesn't export setuptools related question

Assuming you're following the tutorial, you likely installed those libraries EXCEPT for setuptools to lib.

I added setuptools=={verionnumber} to requirements.txt and that fixed this related issue for me.

Answer 3

The google-cloud library is not working on App Engine and most likely you don't even have to since you can use the build in functionality.

From the official docs you can use it like this:

import cloudstorage as gcs