GAE Golang - urlfetch timeout?

Question

I am having issues with urlfetch's timeouts on Google App Engine in Go. The app does not appear to want to take a longer timeout than about 5 seconds (it ignores a longer timeout and times out after its own time).

My code is:

var TimeoutDuration time.Duration = time.Second*30

func Call(c appengine.Context, address string, allowInvalidServerCertificate bool, method string, id interface{}, params []interface{})(map[string]interface{}, error){
    data, err := json.Marshal(map[string]interface{}{
        "method": method,
        "id":     id,
        "params": params,
    })
    if err != nil {
        return nil, err
    }

    req, err:=http.NewRequest("POST", address, strings.NewReader(string(data)))
    if err!=nil{
        return nil, err
    }

    tr := &urlfetch.Transport{Context: c, Deadline: TimeoutDuration, AllowInvalidServerCertificate: allowInvalidServerCertificate}

    resp, err:=tr.RoundTrip(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    result := make(map[string]interface{})
    err = json.Unmarshal(body, &result)
    if err != nil {
        return nil, err
    }
    return result, nil
}

No matter what I try to set TimeoutDuration to, the app times out after about 5 seconds. How prevent it from doing that? Did I make some error in my code?

Answer 1

You need to pass the time duration like this (otherwise it will default to the 5 sec timeout):

tr := &urlfetch.Transport{Context: c, Deadline: time.Duration(30) * time.Second}

Update Jan 2 2016:

With the new GAE golang packages (google.golang.org/appengine/*), this has changed. urlfetch no longer receives a deadline time duration in the transport.

You should now set the timeout via the new context package. For example, this is how you would set a 1 minute deadline:

func someFunc(ctx context.Context) {
    ctx_with_deadline, _ := context.WithTimeout(ctx, 1*time.Minute)
    client := &http.Client{
        Transport: &oauth2.Transport{
            Base:   &urlfetch.Transport{Context: ctx_with_deadline},
        },
    }

Answer 2

This is had now changed with the recent updates to the library . Now the Duration of timeout/delay have to carried by the context , urlfetch.transport no more has the Deadline field in it . context.WithTimeout or context.WithDeadline is the method to use , here is the link https://godoc.org/golang.org/x/net/context#WithTimeout

Answer 3

Try the code below:

// createClient is urlfetch.Client with Deadline
func createClient(context appengine.Context, t time.Duration) *http.Client {
    return &http.Client{
        Transport: &urlfetch.Transport{
            Context:  context,
            Deadline: t,
        },
    }
}

Here is how to use it.

// urlfetch
client := createClient(c, time.Second*60)

Courtesy @gosharplite

Answer 4

Looking at the source code of Go's appengine:

• http://code.google.com/p/appengine-go/source/browse/appengine/urlfetch/urlfetch.go

and the protobuffer generated code:

• http://code.google.com/p/appengine-go/source/browse/appengine_internal/urlfetch/urlfetch_service.pb.go

Looks like there should not be a problem with Duration itself.

My guess is that the whole application inside appengine timeouts after 5 seconds.

Answer 5

for me, this worked:

ctx_with_deadline, _ := context.WithTimeout(ctx, 15*time.Second)
client := urlfetch.Client(ctx_with_deadline)