How to backup an AppEngine site?

Question

So, you build a great shiny cloudy 2.0 website on top of AppEngine, with thousands upon thousands of images saved into the datastore and gigs of data at the blobstore. How do you backup them?

Answer 1

There is now a backup option available in the dashboard. See "datastore admin".

Answer 2

I had some trouble with the suggested solution, so I had to fight with it a little. Here's what I came up with:
1. Add remote_api to app.yaml as described above or simply as
builtins:
- remote_api: on
2. Creating bulk loader as described above resulted in an authentication error for me like bug1125. The command below worked
appcfg.py create_bulkloader_config --filename=<appfolder>/bulkloader.yaml <appfolder>/
3. Change all the TODOs in bulkloader. In my case I changed all the connectors to "csv" and gave all foreign keys meaningful names.
4. I used the following (bash) command to back up all the kinds
for f in cat <appfolder>/bulkloader.yaml | grep "\- kind" | awk '{print $3}'
do
    appcfg.py download_data <appfolder>/ --filename=backup/$f.csv --config_file=<appfolder>/bulkloader.yaml --kind=$f
done


Note: Both commands are meant to run for a folder above the app folder

Answer 3

The command here does not work

http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_and_Uploading_All_Data

--dump needs to be replaced with download_data, --restore needs to be replaced with upload_data and --app_id needs to be changed to --application

Then it would read

Downloading and Uploading All Data

You can download and upload every entity of a kind in a format suitable for backup and restore, all without writing any additional code or configuration. To download all entities of all kinds, run the folowing command:

appcfg.py download_data --application=<app-id> --url=http://<appname>.appspot.com/remote_api --filename=<data-filename>

You can also use the --kind=... argument to download all entities of a specific kind:

appcfg.py download_data --application=<app-id> --kind=<kind> --url=http://<appname>.appspot.com/remote_api --filename=<data-filename>

Note: Downloading all entities of all kinds only works on App Engine, and does not work with the development server. To upload data to the app's datastore from a file created by appcfg.py --dump, run the following command:

appcfg.py upload_data --application=<app-id> --kind=<kind> --filename=<data-filename> <app-directory>

When data is downloaded, the entities are stored along with their original keys. When the data is restored, the original keys are used. If an entity exists in the datastore with the same key as an entity being restored, the entity in the datastore is replaced. You can use --restore to replace the data in the app from which it was dumped, or you can use it to upload the data to a different application. Entities with numeric system IDs will be restored with the same IDs, and reference properties will be preserved.

Answer 4

use google app engine data export http://code.google.com/appengine/docs/python/tools/uploadingdata.html