Undocumented max length for urlfetch URL?

Question

I have an App Engine app that tries to fetch a long url. After experimenting it seems that for some reason App Engine limits the length of fetched urls to 2047. This is the stack trace for the error:

File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 270, in fetch
    return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
    return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 386, in _get_fetch_result
    'Invalid request URL: ' + url + error_detail)
InvalidURLError: Invalid request URL: <LONG URL>

The limit also applies in the development env. where I found the following code in the App Engine sdk (urlfetch_stub.py):

_MAX_URL_LENGTH = 2048

if len(request.url()) >= _MAX_URL_LENGTH:
  logging.error('URL is too long: %s...' % request.url()[:50])
  raise apiproxy_errors.ApplicationError(
      urlfetch_service_pb.URLFetchServiceError.INVALID_URL)

Why does this limit exist? Is there a way to bypass it? It is really critical for me to fetch very long urls (I'm using a REST API service that require long urls).

Answer 1

This is an already reported issue which may be addressed in future. So I guess you have to just wait for now, or use some form of URL shortening (yuck).

You can see the issue here and the related stack question here.

In short, that limit was added to the stub to match the implementation on GAE so at least you found this issue quickly. I have no idea why it is limited.