Cannot deploy App Engine Python 3 after cleanup Cl

Question

I was able to deploy app engine (python 3 standard environment) before with gcloud command. But a few days ago I accidentally removed all images under Cloud Storage (gs://us.artifacts.[My-App-ID].appspot.com/containers/images) and I can no longer deploy new apps to this app-id.

Is there any way I can get back all the files under containers/images? Thanks!

Here is the log obtained from Cloud Build details:

starting build "a71fe78e-463c-43a5-a463-b7c30ce2331f"

FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetching manifest gs://staging.[My-App-ID].appspot.com/ae/357626f2-3c12-444a-bf54-cb0d488511b8/manifest.json.
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/ae/357626f2-3c12-444a-bf54-cb0d488511b8/manifest.json (3340B in 267.326596ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Processing 16 files.
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/ccd9f8893da70ab46e98841ee844d2cb848b4a6d (439B in 50.73556ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/5ea48c0d84a0026d4dd85572f135bdeeb4a8658f (485B in 75.680668ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/a508e33e1c6b091ce6168f8f1420481fb527cb62 (591B in 77.662409ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/223ddac1a197106107aedeba74caa799620393a1 (726B in 81.764857ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/11e72a955893ccfc1205a8cddbc0a4a4865aa49e (99B in 83.761955ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/68d5bda667cc4702b7637fc684da9e835807986d (116B in 86.414682ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/bd8a82fd1ddc2268d5b748436ef5bbd5a52f526a (420B in 89.246942ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/d0e86fe12323f4f235dbdae8e7406f254f9bae0e (6755B in 91.486346ms, 0.07MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/6d360bdd54449a7226efe3c8aafb1a8941340930 (249B in 95.961089ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/cb6f5d5a43aa064eceed3229b45502eff5ff637a (344B in 99.508753ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/b0e01ca8cc57794f92acc78be415e074ded73f30 (747B in 99.025703ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/03fb2be08403bc11bc9af9b4a782e4a163bc8a51 (8573B in 106.974844ms, 0.08MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/ce74e53c14f814f65dc339c8a376e1c3bab557f1 (712B in 108.83968ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/56d28df9bfffa6cfac74e676ffeb769dedafe29c (340B in 112.297078ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/78683652584e785421f4e3a624084e5ad4169c2f (1172B in 116.225539ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 Fetched gs://staging.[My-App-ID].appspot.com/11886d6f17bc4df6737966ccef8e4a4f281fa195 (4430B in 130.78125ms, 0.03MiB/s)
Step #0 - "fetcher": 2018/09/05 03:22:49 ******************************************************
Step #0 - "fetcher": 2018/09/05 03:22:49 Status: SUCCESS
Step #0 - "fetcher": 2018/09/05 03:22:49 Started: 2018-09-05T03:22:49Z
Step #0 - "fetcher": 2018/09/05 03:22:49 Completed: 2018-09-05T03:22:49Z
Step #0 - "fetcher": 2018/09/05 03:22:49 Requested workers: 200
Step #0 - "fetcher": 2018/09/05 03:22:49 Actual workers: 16
Step #0 - "fetcher": 2018/09/05 03:22:49 Total files: 16
Step #0 - "fetcher": 2018/09/05 03:22:49 Total retries: 0
Step #0 - "fetcher": 2018/09/05 03:22:49 GCS timeouts: 0
Step #0 - "fetcher": 2018/09/05 03:22:49 MiB downloaded: 0.02 MiB
Step #0 - "fetcher": 2018/09/05 03:22:49 MiB/s throughput: 0.19 MiB/s
Step #0 - "fetcher": 2018/09/05 03:22:49 Time for manifest: 267.33 ms
Step #0 - "fetcher": 2018/09/05 03:22:49 Total time: 0.40 s
Step #0 - "fetcher": 2018/09/05 03:22:49 ******************************************************
Finished Step #0 - "fetcher"
Starting Step #1 - "builder"
Step #1 - "builder": Pulling image: gcr.io/gae-runtimes/python37_app_builder:python37_3_7_0_20180820_RC00
Step #1 - "builder": python37_3_7_0_20180820_RC00: Pulling from gae-runtimes/python37_app_builder
Step #1 - "builder": Digest: sha256:94db1bd24e2ec82ab822a6cc44509a3a1579bb6a3fb8a3cdfe203f2f82f6b03e
Step #1 - "builder": Status: Downloaded newer image for gcr.io/gae-runtimes/python37_app_builder:python37_3_7_0_20180820_RC00
Step #1 - "builder": 05 Sep 2018 03:22:51 INFO Arguments: ['--parser_script=/ftl-v0.7.0.par', '--name=us.gcr.io/[My-App-ID]/app-engine/default/20180905-staging-6020:357626f2-3c12-444a-bf54-cb0d488511b8', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/[My-App-ID]/app-engine-build-cache', '--base=gcr.io/gae-runtimes/python37:python37_3_7_0_20180820_RC00', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--virtualenv-cmd=/opt/python3.7/bin/python3.7 -m virtualenv', '-v=DEBUG', '--entrypoint-from-app-yaml=true', '--entrypoint-contents=']
Step #1 - "builder": 05 Sep 2018 03:22:51 INFO Unparsed arguments: ['--name=us.gcr.io/[My-App-ID]/app-engine/default/20180905-staging-6020:357626f2-3c12-444a-bf54-cb0d488511b8', '--destination=/srv', '--cache-repository=us.gcr.io/[My-App-ID]/app-engine-build-cache', '--base=gcr.io/gae-runtimes/python37:python37_3_7_0_20180820_RC00', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--virtualenv-cmd=/opt/python3.7/bin/python3.7 -m virtualenv', '-v=DEBUG']
Step #1 - "builder": 05 Sep 2018 03:22:51 INFO Checking app.yaml for entrypoint information
Step #1 - "builder": 05 Sep 2018 03:22:51 INFO Entrypoint: {'type': 'default'}
Step #1 - "builder": 05 Sep 2018 03:22:51 INFO Executing ['/ftl-v0.7.0.par', '--name=us.gcr.io/[My-App-ID]/app-engine/default/20180905-staging-6020:357626f2-3c12-444a-bf54-cb0d488511b8', '--destination=/srv', '--cache-repository=us.gcr.io/[My-App-ID]/app-engine-build-cache', '--base=gcr.io/gae-runtimes/python37:python37_3_7_0_20180820_RC00', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--virtualenv-cmd=/opt/python3.7/bin/python3.7 -m virtualenv', '-v=DEBUG', '--entrypoint=/start', '--directory=/workspace', '--additional-directory=/.gaeconfig']
Step #1 - "builder": INFO FTL version python-v0.7.0
Step #1 - "builder": INFO Beginning FTL build for python
Step #1 - "builder": INFO FTL arg passed: exposed_ports None
Step #1 - "builder": INFO FTL arg passed: cache_repository us.gcr.io/[My-App-ID]/app-engine-build-cache
Step #1 - "builder": INFO FTL arg passed: tar_base_image_path None
Step #1 - "builder": INFO FTL arg passed: builder_output_path /builder/outputs
Step #1 - "builder": INFO FTL arg passed: name us.gcr.io/[My-App-ID]/app-engine/default/20180905-staging-6020:357626f2-3c12-444a-bf54-cb0d488511b8
Step #1 - "builder": INFO FTL arg passed: pip_cmd /env/bin/python3.7 -m pip
Step #1 - "builder": INFO FTL arg passed: venv_dir /env
Step #1 - "builder": INFO FTL arg passed: global_cache False
Step #1 - "builder": INFO FTL arg passed: cache True
Step #1 - "builder": INFO FTL arg passed: upload True
Step #1 - "builder": INFO FTL arg passed: sh_c_prefix False
Step #1 - "builder": INFO FTL arg passed: fail_on_error True
Step #1 - "builder": INFO FTL arg passed: base gcr.io/gae-runtimes/python37:python37_3_7_0_20180820_RC00
Step #1 - "builder": INFO FTL arg passed: output_path None
Step #1 - "builder": INFO FTL arg passed: directory /workspace
Step #1 - "builder": INFO FTL arg passed: entrypoint /start
Step #1 - "builder": INFO FTL arg passed: additional_directory /.gaeconfig
Step #1 - "builder": INFO FTL arg passed: destination_path /srv
Step #1 - "builder": INFO FTL arg passed: verbosity DEBUG
Step #1 - "builder": INFO FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m virtualenv
Step #1 - "builder": INFO FTL arg passed: python_cmd /opt/python3.7/bin/python3.7
Step #1 - "builder": INFO starting: full build
Step #1 - "builder": INFO starting: builder initialization
Step #1 - "builder": INFO Loading Docker credentials for repository 'gcr.io/gae-runtimes/python37:python37_3_7_0_20180820_RC00'
Step #1 - "builder": INFO Loading Docker credentials for repository 'us.gcr.io/[My-App-ID]/app-engine/default/20180905-staging-6020:357626f2-3c12-444a-bf54-cb0d488511b8'
Step #1 - "builder": INFO builder initialization took 0 seconds
Step #1 - "builder": INFO starting: build process for FTL image
Step #1 - "builder": INFO starting: checking_cached_interpreter_layer
Step #1 - "builder": INFO starting: check python version
Step #1 - "builder": INFO `python version` full cmd:
Step #1 - "builder": /opt/python3.7/bin/python3.7 --version
Step #1 - "builder": INFO `python version` stderr:
Step #1 - "builder": 
Step #1 - "builder": INFO check python version took 0 seconds
Step #1 - "builder": DEBUG Checking cache for cache_key 3bd46fc17b674007635ca68c1879e947eb7ed488a1f24958cecd85bc66b56218
Step #1 - "builder": INFO Found cached base image: us.gcr.io/[My-App-ID]/app-engine-build-cache/python-cache:3bd46fc17b674007635ca68c1879e947eb7ed488a1f24958cecd85bc66b56218.
Step #1 - "builder": INFO Found cached dependency layer for 3bd46fc17b674007635ca68c1879e947eb7ed488a1f24958cecd85bc66b56218
Step #1 - "builder": INFO checking_cached_interpreter_layer took 0 seconds
Step #1 - "builder": INFO build process for FTL image took 0 seconds
Step #1 - "builder": INFO full build took 0 seconds
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder": exec code in run_globals
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__.py", line 65, in <module>
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__.py", line 54, in main
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__/ftl/python/builder.py", line 68, in Build
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__/ftl/python/layer_builder.py", line 332, in BuildLayer
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__/ftl/common/cache.py", line 115, in Get
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__/ftl/common/cache.py", line 178, in checkTTL
Step #1 - "builder": File "/ftl-v0.7.0.par/__main__/ftl/common/ftl_util.py", line 186, in creation_time
Step #1 - "builder": File "/ftl-v0.7.0.par/containerregistry/client/v2_2/docker_image_.py", line 306, in config_file
Step #1 - "builder": File "/ftl-v0.7.0.par/containerregistry/client/v2_2/docker_image_.py", line 330, in blob
Step #1 - "builder": File "/ftl-v0.7.0.par/containerregistry/client/v2_2/docker_image_.py", line 250, in _content
Step #1 - "builder": File "/ftl-v0.7.0.par/containerregistry/client/v2_2/docker_http_.py", line 364, in Request
Step #1 - "builder": containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '404', 'content-length': '290', 'expires': 'Wed, 05 Sep 2018 03:22:52 GMT', 'server': 'UploadServer', 'cache-control': 'private, max-age=0', 'date': 'Wed, 05 Sep 2018 03:22:52 GMT', 'content-type': 'application/xml; charset=UTF-8', 'x-guploader-uploadid': 'AEnB2Upug586B6cmKZxDUYGPgvMDvdXfoBqlnrWXkN7qwb4Q3oCMA9DnvbPnJdc7IaEez9mrzB3KAPzSFF5vWyRIJozRMW5v4ehHrq3TL8K8bjPAuuOF4Uk'}
Step #1 - "builder": <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.[My-App-ID].appspot.com/containers/images/sha256:3ac54797dbe38d0f8c7684c3ce9273d79530b3c724616c38270e0692ed2c900b</Details></Error>: None
Finished Step #1 - "builder"
ERROR
ERROR: build step 1 "gcr.io/gae-runtimes/python37_app_builder:python37_3_7_0_20180820_RC00" failed: exit status 1

Answer 1

You seem to have deleted the container image artifacts stored in Google Cloud Storage, however the image layers probably still existing in Container Registry.

Deleting the images in Container Registry should fix the situation:

1. Open the Container Registry (https://console.cloud.google.com/gcr/)

2. Manually delete all images stored in the bucket us.gcr.io/[My-App-ID]/app-engine-build-cache/python-cache.