App Engine: Alternatives to urlfetch? Seems very u

Question

I'm using urlfetch in my app and while everything works perfectly fine in the development environment, i'm finding urlfetch to be VERY unreliable when it's actually deployed. Sometimes it works as it should (retrieving data), but then a few minutes later it might return nothing, then it'll be working fine again a few minutes after that. This is very unacceptable. I've checked to make sure it's NOT the source URL that's the problem (YQL) and, again, everything works as it should in the development environment.

Are there any third-party libraries I could try?

Example code:

url = "http://query.yahooapis.com/v1/public/yql?q=%s&format=json" % urllib.quote_plus(query)
result = urlfetch.fetch(url, deadline=10)

if result.status_code == 200:
    r = json.loads(result.content)
else:
    return

a = r['query']['results']
# Do stuff with 'a'

Sometimes it'll work as it should, but other times - completely randomly with no code changes - i'll get this this error:

a = r['query']['results']
TypeError: 'NoneType' object is unsubscriptable

Answer 1

import urllib2
response = urllib2.urlopen('http://python.org/')
html = response.read()

Answer 2

Sometimes it'll work as it should, but other times completely randomly with no code changes

This is a common symptom that your application's requests have exceeded the Yahoo API calls rate limit.

Quoting Yahoo developer documentations rate limit:

IP Based Limits

Our service rate limits are imposed as a limit on the number of API calls made per IP address during a specific time window. If your IP address changes during that time period, you may find yourself with more "credit" available. However, if someone else had been using the address and hit the limit, you'll need to wait until the end of the time period to be allowed to make more API calls.

Google App Engine uses a pool of IP addresses for outgoing urlfetch requests and your application is sharing these IP addresses with other applications that are calling the same Yahoo endpoint; when the rate limit is exceeded, the endpoint replies with a limit exceeded error causing UrlFetch to fail.
Here another case using the Twitter search API.

When you mix Google App Engine+Third party web APIs, you need to be sure that the API provides authenticated calls allowing your application to have its own quota (StackApps API for example).

Answer 3

This isn't an error in URLFetch - it's an issue with the JSON being returned. Either json.loads is returning None, or r['query'] is - I'm guessing it's probably the latter. Try logging result.content to see what the service is returning. You probably also want to cehck result.status.

One possibility is that your request is being denied or ratelimited by Yahoo in production, but not on your development machine.