app engine for python on ubuntu import datastore l

Question

I'm trying to create a local copy of the datastore following the answer on this question How to create local copy of GAE datastore? . On MAC/Windows it works, but now i'm using Ubuntu and I get this error:

Traceback (most recent call last):
  File "/opt/google/google_appengine/appcfg.py", line 133, in <module>
    run_file(__file__, globals())
  File "/opt/google/google_appengine/appcfg.py", line 129, in run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 5445, in <module>
    main(sys.argv)
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 5436, in main
    result = AppCfgApp(argv).Run()
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 2997, in Run
    self.action(self)
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 5092, in __call__
    return method()
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 4874, in PerformDownload
    run_fn(args)
  File "/opt/google/google_appengine/google/appengine/tools/appcfg.py", line 4777, in RunBulkloader
    sys.exit(bulkloader.Run(arg_dict))
  File "/opt/google/google_appengine/google/appengine/tools/bulkloader.py", line 4405, in Run
    return _PerformBulkload(arg_dict)
  File "/opt/google/google_appengine/google/appengine/tools/bulkloader.py", line 4145, in _PerformBulkload
    passin = arg_dict['passin']
KeyError: 'passin'

I tried using --passin, but there is no such argument to appcfg.py Any ideas about what could go wrong?

The command I am trying to use is:

appcfg.py upload_data --filename=Downloads/data.csv --url=http://localhost:8080/remote_api

Answer 1

Just tried setting passin = False instead of getting it from arg_dict['passin'] in bulkloader.py and it works.

passin = False #arg_dict['passin']
self.passin = False #arg_dict['passin']

Answer 2

I managed to upload data to the local development server. I am using the gcloud 0.9.85 with app-engine SDK 1.9.28.

I start the server with the following command:

gcloud preview app run app.yaml --host 0.0.0.0:8080

It will start the API server at some random port (e.g. 64578). You can fix the port with --api-host anyway.

To upload the data to the local datastore:

appcfg.py upload_data --application=dev~APPLICATION_ID --url=http://localhost:API_SERVER_PORT/_ah/remote_api --filename=DB_DOWNLOAD_FROM_APPSPOT

Answer 3

You can also use bulkloader.py instead of appcfg.py, as documented here. In a forthcoming SDK release, appcfg upload/download_data will work again, but it will use OAuth2.

Answer 4

The recipe you're trying might no longer work with (or may need to be updated for) the more recent versions of the SDK since the plain password based authentication in appcfg.py was dropped. From the SDK release notes for for 1.9.24:

In all the App Engine SDKs, authentication for app deployment is now exclusively through OAuth2. Authentication using an email address and password is no longer supported and the --no_oauth2 flag is no longer available. Note that email address/password authentication will also soon cease to work for older SDK versions.