Migrating from App Engine Files API

Question

My app stores a bunch of images as blobs. This is roughly how I store images.

from google.appengine.api import files
# ...
fname = files.blobstore.create(mime_type='image/jpeg')
with files.open(fname, 'a') as f:
  f.write(image_byte)
files.finalize(fname)
blob_key = files.blobstore.get_blob_key(fname)

To serve these images, I use images.get_serving_url(blob_key).

Here are my questions:

1. Will I have to copy over all blobs to Google Cloud Storage? In other words, will I be able to access my existing blobs using GCS client library and existing blob keys? Or, will I have to copy the blobs over to GCS and get new blob keys?
2. Assuming I do have to copy them over to GCS, what is the easiest way? Is there a migration tool or something? Failing that, is there some sample code I can copy-paste?

Thanks!

Answer 1

The files have all been going into GCS for a while. The blobstore is just an alternate way to access it. The blob keys and access shouldn't be affected.

You will, however, need to stop using the files API itself and start using the GCS API to create the files.

Answer 2

I received an email from Google Cloud Platform on May 19, 2015, an excerpt is shown here:

The removal of the Files API will happen in the following manner.

On May 20th, 2015 no new applications will have access to the Files API. Applications that were created prior to May 20th, 2015 will continue to run without any issues. That said, we strongly encourage developers to start switching over to the Cloud Storage Client Library today.

On July 28th, 2015 starting at 12pm Pacific Time, the Files API will be temporarily shutdown for 24 hrs.

On August 4th, 2015, we will permanently shut down the Files API at 12:00pm Pacific time.

Since I was using the exact same code to write a blobstore file, I spent a day researching the GCS system. After failing to get a "service account" to work (by going through poorly documented OAuth2 confusion), I gave up on using GCS.

Now I am using ndb's BlobProperty. I keep the blobs in a separate model using both a parent key and a key name (as filename) to locate the images. Using a separate model keeps the huge blob out of my regular entities so fetches aren't slowed down by their sheer size. I wrote a separate REST API just for the images.

Answer 3

Me too faced same issue while running GAE server locally:

com.google.appengine.tools.cloudstorage.NonRetriableException: com.google.apphosting.api.ApiProxy$FeatureNotEnabledException: The Files API is disabled. Further information: https://cloud.google.com/appengine/docs/deprecations/files_api

Here in my case this is fixed my issue:

Simply I changed

This:

compile 'com.google.appengine.tools:appengine-gcs-client:0.4.1'

To:

compile 'com.google.appengine.tools:appengine-gcs-client:0.5'

in build.gradle file, because Files API(Beta) is deprecaated on June 12, 2013 and Turndowned on September 9, 2015. (Source)

From this MVN Repo latest one is 'com.google.appengine.tools:appengine-gcs-client:0.5'

Answer 4

1) No, you can still use the blobstore. You can also upload files to the blobstore when you use the BlobstoreUploadHandler.

2) Migration is easy when you use the blobstore, bacause you can create a blobkey for GCS objects. And when you use the default GCS bucket you have free quota.

from google.appengine.api import app_identity
import cloudstorage as gcs

default_bucket = app_identity.get_default_gcs_bucket_name() 
gcs_filename = '/%s/%s' % (default_bucket, image_file_name)
with gcs.open(gcs_filename, 'w', content_type='image/jpeg') as f:
    f.write(image_byte)

blob_key = blobstore.create_gs_key('/gs' + gcs_filename)
# and create a serving url