Need Help Creating GAE Datastore Loader Class?

Question

Need Help Creating GAE Datastore Loader Class for uploading data using appcfg.py? Any other way to simplified this process? is there any detailed example better than here

When try using bulkloader.yaml:

Uploading data records.
[INFO    ] Logging to bulkloader-log-20100701.041515
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20100701.041515.sql3
[INFO    ] Connecting to livelihoodproducer.appspot.com/remote_api
[INFO    ] Starting import; maximum 10 entities per post
[ERROR   ] [Thread-1] WorkerThread:
Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
    status, instruction = item.PerformWork(self.__thread_pool)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork
    transfer_time = self._TransferItem(thread_pool)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem
    self.content = self.request_manager.EncodeContent(self.rows)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent
    entity = loader.create_entity(values, key_name=key, parent=parent)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity
    return self.dict_to_entity(input_dict, self.bulkload_state)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 233, in __run_import_transforms
    value = self.__dict_to_prop(transform, input_dict, bulkload_state)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop
    value = transform.import_transform(value)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__
    return self.method(*args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 143, in generate_foreign_key_lambda
    return datastore.Key.from_path(kind, value)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 387, in from_path
    'received %r (a %s).' % (i + 2, id_or_name, typename(id_or_name)))
BadArgumentError: Expected an integer id or string name as argument 2; received None (a NoneType).
[INFO    ] [Thread-3] Backing off due to errors: 1.0 seconds
[INFO    ] Unexpected thread death: Thread-1
[INFO    ] An error occurred. Shutting down...
[ERROR   ] Error in Thread-1: Expected an integer id or string name as argument 2; received None (a NoneType).

[INFO    ] 30 entites total, 0 previously transferred
[INFO    ] 0 entities (733 bytes) transferred in 2.8 seconds
[INFO    ] Some entities not successfully transferred

In the process, i've downloadeded csv data manually inserted on appspot.com. While i try to upload my own csv data, the column order should made exactly like csv downloaded from appspot.com? how about blank value?

Answer 1

I've created config.yaml with bulkloader config, and also written simple helper function to process None-references. I don't know why it's not done in original helper.

The helper (file helpers.py is very simple, just place it to the same directory where you placed config.yaml):

from google.appengine.api import datastore
def create_foreign_key(kind, key_is_id=False):
  def generate_foreign_key_lambda(value):
    if value is None:
      return None

    if key_is_id:
      value = int(value)
    return datastore.Key.from_path(kind, value)

  return generate_foreign_key_lambda

And this is cut from my config.yaml:

python_preamble:
- import: helpers # this will import our helper
[other imports]
...
- kind: ArticleComment
  connector: simplexml
  connector_options:
    xpath_to_nodes: "/blog/Comments/Comment"
    style: element_centric

  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: parent_comment
      external_name: parent-comment
      export_transform: transform.key_id_or_name_as_string
      import_transform: helpers.create_foreign_key('ArticleComment')
      #                 ^^^^^^^ here it is
      #                 use this instead of transform.create_foreign_key

Answer 2

It looks like you have reference properties with None values, such values are handled incorrectly by bulkloader's helpers.