golang http timeout and goroutines accumulation

Question

I use goroutines achieve http.Get timeout, and then I found that the number has been rising steadily goroutines, and when it reaches 1000 or so, the program will exit

Code:

package main

import (
        "errors"
        "io/ioutil"
        "log"
        "net"
        "net/http"
        "runtime"
        "time"
)

// timeout dialler
func timeoutDialler(timeout time.Duration) func(network, addr string) (net.Conn, error) {
        return func(network, addr string) (net.Conn, error) {
                return net.DialTimeout(network, addr, timeout)
        }
}

func timeoutHttpGet(url string) ([]byte, error) {
        // change dialler add timeout support && disable keep-alive
        tr := &http.Transport{
                Dial:              timeoutDialler(3 * time.Second),
                DisableKeepAlives: true,
        }

        client := &http.Client{Transport: tr}

        type Response struct {
                resp []byte
                err  error
        }

        ch := make(chan Response, 0)
        defer func() {
                close(ch)
                ch = nil
        }()

        go func() {
                resp, err := client.Get(url)
                if err != nil {
                        ch <- Response{[]byte{}, err}
                        return
                }
                defer resp.Body.Close()

                body, err := ioutil.ReadAll(resp.Body)
                if err != nil {
                        ch <- Response{[]byte{}, err}
                        return
                }

                tr.CloseIdleConnections()
                ch <- Response{body, err}
        }()

        select {
        case <-time.After(5 * time.Second):
                return []byte{}, errors.New("timeout")
        case response := <-ch:
                return response.resp, response.err
        }
}

func handler(w http.ResponseWriter, r *http.Request) {
        _, err := timeoutHttpGet("http://google.com")
        if err != nil {
                log.Println(err)
                return
        }
}

func main() {
        go func() {
                for {
                        log.Println(runtime.NumGoroutine())
                        time.Sleep(500 * time.Millisecond)
                }
        }()

        s := &http.Server{
                Addr:         ":8888",
                ReadTimeout:  15 * time.Second,
                WriteTimeout: 15 * time.Second,
        }

        http.HandleFunc("/", handler)
        log.Fatal(s.ListenAndServe())
}

http://play.golang.org/p/SzGTMMmZkI

Answer 1

Init your chan with 1 instead of 0:

ch := make(chan Response, 1)

And remove the defer block that closes and nils ch.

See: http://blog.golang.org/go-concurrency-patterns-timing-out-and

Here is what I think is happening:

1. after the 5s timeout, timeoutHttpGet returns
2. the defer statement runs, closing ch and then setting it to nil
3. the go routine it started to do the actual fetch finishes and attempts to send its data to ch
4. but ch is nil, and so won't receive anything, preventing that statement from finishing, and thus preventing the go routine from finishing

I assume you are setting ch = nil because before you had that, you would get run-time panics because that's what happens when you attempt to write to a closed channel, as described by the spec.

Giving ch a buffer of 1 means that the fetch go routine can send to it without needing a receiver. If the handler has returned due to timeout, everything will just get garbage collected later on.