云机器学习引擎运转时Tensorflow物体检测train.py失败(Tensorflow obje

我有tensorflow物体检测API本地工作的一个小的工作示例。一切看起来都很好。我的目标是用自己的脚本在谷歌机器学习引擎，我已经在过去广泛使用运行。我下面这些文档。

声明一些相关变量

declare PROJECT=$(gcloud config list project --format "value(core.project)")
declare BUCKET="gs://${PROJECT}-ml"
declare MODEL_NAME="DeepMeerkatDetection"
declare FOLDER="${BUCKET}/${MODEL_NAME}"
declare JOB_ID="${MODEL_NAME}_$(date +%Y%m%d_%H%M%S)"
declare TRAIN_DIR="${FOLDER}/${JOB_ID}"
declare EVAL_DIR="${BUCKET}/${MODEL_NAME}/${JOB_ID}_eval"
declare  PIPELINE_CONFIG_PATH="${FOLDER}/faster_rcnn_inception_resnet_v2_atrous_coco_cloud.config"
declare  PIPELINE_YAML="/Users/Ben/Documents/DeepMeerkat/training/Detection/cloud.yml"

我YAML看起来像

trainingInput:
  runtimeVersion: "1.0"
  scaleTier: CUSTOM
  masterType: standard_gpu
  workerCount: 5
  workerType: standard_gpu
  parameterServerCount: 3
  parameterServerType: standard

相关的路径在配置，如设置

  fine_tune_checkpoint: "gs://api-project-773889352370-ml/DeepMeerkatDetection/checkpoint/faster_rcnn_inception_resnet_v2_atrous_coco_11_06_2017/model.ckpt"

我已经打包的对象检测和使用超薄setup.py

运行

gcloud ml-engine jobs submit training "${JOB_ID}_train" \
    --job-dir=${TRAIN_DIR} \
    --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz \
    --module-name object_detection.train \
    --region us-central1 \
    --config ${PIPELINE_YAML} \
    -- \
    --train_dir=${TRAIN_DIR} \
    --pipeline_config_path= ${PIPELINE_CONFIG_PATH}

产生一个tensorflow（进口？）错误。它有点神秘

insertId:  "1inuq6gg27fxnkc"  
 logName:  "projects/api-project-773889352370/logs/ml.googleapis.com%2FDeepMeerkatDetection_20171017_141321_train"  
 receiveTimestamp:  "2017-10-17T21:38:34.435293164Z"  
 resource: {…}  
 severity:  "ERROR"  
 textPayload:  "The replica ps 0 exited with a non-zero status of 1. Termination reason: Error. 
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 145, in main
    model_config, train_config, input_config = get_configs_from_multiple_files()
  File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 127, in get_configs_from_multiple_files
    text_format.Merge(f.read(), train_config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 112, in read
    return pywrap_tensorflow.ReadFromStream(self._read_buf, length, status)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
FailedPreconditionError: .

我已经看到了在其他此错误的问题涉及到预测的机器学习引擎，可能提示这个错误（？）不直接与对象相关的检测代码，但感觉像其没有被正确包装，缺少的依赖？我已经更新了我gcloud到最新版本。

Bens-MacBook-Pro:research ben$ gcloud --version
Google Cloud SDK 175.0.0
bq 2.0.27
core 2017.10.09
gcloud 
gsutil 4.27

很难看出它与此相关的问题就在这里

FailedPreconditionError运行有自己的模型TF目标检测API时

为什么代码需要在云中初始化的不同？

更新＃1。

好奇的一点是，eval.py正常工作，所以它不能是配置文件，或任何train.py和eval.py共享路径。 Eval.py耐心地坐着要创建的模型检查站等待。

另一个想法可能是，检查点莫名其妙地被上传过程中被损坏。我们可以测试从头这个旁路和培训。

在的.config

  from_detection_checkpoint: false

这产生了相同的前提错误，所以它不可能是模型。

根本原因是轻微的错字：

--pipeline_config_path= ${PIPELINE_CONFIG_PATH}

有一个额外的空间。尝试这个：

gcloud ml-engine jobs submit training "${JOB_ID}_train" \
    --job-dir=${TRAIN_DIR} \
    --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz \
    --module-name object_detection.train \
    --region us-central1 \
    --config ${PIPELINE_YAML} \
    -- \
    --train_dir=${TRAIN_DIR} \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH}