Why is Google Appengine so slow connecting to Clou

I am seeing a drastic difference in latency between development and production when connecting to a CloudSQL backend, much more so than I would expect.

I ran a test where:

I fetched 125, 250, 500, 1000 and 2000 rows (row size approximately 30bytes)
I fetched each row size 20 times, to get a good sampling of the time

The test was run in three environments:

Hosted appengine
Development mode locally, but connecting to CloudSQL via static IP
Development mode locally and connecting to a local VM running MySQL

Here you can see the results:

Now I would expect some speed fluctuations on the order of 50ms-200ms but 3-4 seconds seems a bit high.

I'm new to appengine, so any newb mistakes that might be causing this? Or other suggestions? I ran a profiler on my code in appengine and there is a call to _apiProxy.Event "wait" that eats up at least 500ms, but didn't go up more than 750ms, other than that, there was any long running calls. A number of shorter running calls that eventually add up of course, but it's not like I have a loop that needs to be tuned or anything.

Thanks in advance!

First off, check the connectivity path you are using: are you connecting via the latest documented method? Cloud SQL used to have a connectivity path which is slower and is now deprecated, but still functioning, so you could be accessing via that.

Second, is the App Engine app and Cloud SQL instance in the same location? Check that the "Preferred Location" in your Cloud SQL settings are set to follow the app engine app you are connecting to.

As a last possibility, which seems unlikely given that you have data connecting locally, make sure you are reusing database connections, making new ones can be expensive. If there was some reason why your app is reusing connections locally but making new ones on the App Engine side, that could create this behavior. But like I said, this one seems unlikely.