Reusing http connections in Golang

问题:

I'm currently struggling to find a way to reuse connections when making HTTP posts in Golang.

I've created a transport and client like so:

// Create a new transport and HTTP client
tr := &http.Transport{}
client := &http.Client{Transport: tr}

I'm then passing this client pointer into a goroutine which is making multiple posts to the same endpoint like so:

r, err := client.Post(url, "application/json", post)

Looking at netstat this appears to be resulting in a new connection for every post resulting in a large number of concurrent connections being open.

What is the correct way to reuse connections in this case?

回答1:

You should ensure that you read until the response is complete before calling Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

To ensure http.Client connection reuse be sure to do two things:

• Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
• Call Body.Close()

回答2:

If anyone is still finding answers on how to do it, this is how I am doing it.

package main

import (
    "bytes"
    "io/ioutil"
    "log"
    "net/http"
    "time"
)

var httpClient *http.Client

const (
    MaxIdleConnections int = 20
    RequestTimeout     int = 5
)

func init() {
    httpClient = createHTTPClient()
}

// createHTTPClient for connection re-use
func createHTTPClient() *http.Client {
    client := &http.Client{
        Transport: &http.Transport{
            MaxIdleConnsPerHost: MaxIdleConnections,
        },
        Timeout: time.Duration(RequestTimeout) * time.Second,
    }

    return client
}

func main() {
    endPoint := "https://localhost:8080/doSomething"

    req, err := http.NewRequest("POST", endPoint, bytes.NewBuffer([]byte("Post this data")))
    if err != nil {
        log.Fatalf("Error Occured. %+v", err)
    }
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

    response, err := httpClient.Do(req)
    if err != nil && response == nil {
        log.Fatalf("Error sending request to API endpoint. %+v", err)
    }

    // Close the connection to reuse it
    defer response.Body.Close()

    // Let's check if the work actually is done
    // We have seen inconsistencies even when we get 200 OK response
    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Fatalf("Couldn't parse response body. %+v", err)
    }

    log.Println("Response Body:", string(body))    
}

Go Playground: http://play.golang.org/p/oliqHLmzSX

In summary, I am creating a different method to create a HTTP client and assigning it to global variable and then using it to make requests. Note the

defer response.Body.Close() 

This will close the connection and set it ready for reuse again.

Hope this will help someone.

回答3:

Edit: This is more of a note for people that construct a Transport and Client for every request.

Edit2: Changed link to godoc.

Transport is the struct that holds connections for re-use; see https://godoc.org/net/http#Transport ("By default, Transport caches connections for future re-use.")

So if you create a new Transport for each request, it will create new connections each time. In this case the solution is to share the one Transport instance between clients.

回答4:

IIRC, the default client does reuse connections. Are you closing the response?

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

回答5:

about Body

// It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.

So if you want to reuse TCP connections, you have to close Body every time after read to completion. An function ReadBody(io.ReadCloser) is suggested like this.

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://github.com", nil)
    if err != nil {
        fmt.Println(err.Error())
        return
    }
    client := &http.Client{}
    i := 0
    for {
        resp, err := client.Do(req)
        if err != nil {
            fmt.Println(err.Error())
            return
        }
        _, _ = readBody(resp.Body)
        fmt.Println("done ", i)
        time.Sleep(5 * time.Second)
    }
}

func readBody(readCloser io.ReadCloser) ([]byte, error) {
    defer readCloser.Close()
    body, err := ioutil.ReadAll(readCloser)
    if err != nil {
        return nil, err
    }
    return body, nil
}

回答6:

There are two possible ways:

1. Use a library that internally reuses and manages the file descriptors, associated with each requests. Http Client does the same thing internally, but then you would have the control over how many concurrent connections to open, and how to manage your resources. If you are interested, look at the netpoll implementation, which internally uses epoll/kqueue to manage them.

2. The easy one would be, instead of pooling network connections, create a worker pool, for your goroutines. This would be easy, and better solution, that would not hinder with your current codebase, and would require minor changes.

Let's assume you need to make n POST request, after you recieve a request.

You could use channels, to implement this.

Or, simply you could use third party libraries.
Like: https://github.com/ivpusic/grpool