Google App Engine Go HTTP request to a slow page

Question

I want to make a HTTP request using Google App Engine in Go to a slow-responding website. My code is:

func request(w http.ResponseWriter, r *http.Request) {
    c:=appengine.NewContext(r)
    client := urlfetch.Client(c)
    resp, err := client.Get("http://www.example.com")
    if err != nil {
        log.Print("err %s", err.String())
        return
    }
    ...}

After calling my website, I get such an error:

API error 1 (urlfetch: INVALID_URL): ApplicationError: 5 timed out

After looking a bi into it, I found some reference to Transport class that allows to wait for HTTP requests for up to 60 seconds, but I can't find any description or example on how to make that work.

How can I make a HTTP request in GAE Go that will have a deadline longer than 5 seconds?

Answer 1

I would recommend if possible, doing the urlfetch outside of your main request handler using Tasks Queue API.

When constructing urlfetch.Transport, you should be able to override the default deadline like this:

t := urlfetch.Transport{Context:c, DeadlineSeconds: 5.0}
client := http.Client{Transport: &t}

Answer 2

Remote sites are sometimes slow because of DNS resolution. Using the IP address might help (resolve the target host before connecting to http://a.b.c.d).

I don't know if Go lets you define a longer timeout value - what you can do is to poll several times until you get a reply (or have expired your retries count).

Good luck anyway.