Why is Go HTTPS Client not reusing connections?

Question

I have an http client which creates multiple connections to the host. I want to set a maximum number of connections it can set to a particular host. There are no such options in go's request.Transport. My code looks like

package main 

import (
  "fmt"
  "net/http"
  "net/url"
)

const (
  endpoint_url_fmt      = "https://blah.com/api1?%s"
)


func main() {

  transport := http.Transport{ DisableKeepAlives : false }

  outParams := url.Values{}
  outParams.Set("method", "write")
  outParams.Set("message", "BLAH")

  for {
    // Encode as part of URI.
    outboundRequest, err := http.NewRequest(
      "GET",
      fmt.Sprintf(endpoint_url_fmt, outParams.Encode()),
      nil
    )
    outboundRequest.Close = false
    _ , err = transport.RoundTrip(outboundRequest)
    if err != nil {
      fmt.Println(err)
    }
  }

}

I would expect this to create 1 connection. As I am calling it in a for-loop. But this keeps creating an infinite number of connections.

Where as similar python code using the requests library creates only one connection.

#!/usr/bin/env python
import requests
endpoint_url_fmt      = "https://something.com/restserver.php"
params = {}
params['method'] = 'write'
params['category'] = category_errors_scuba
params['message'] = "blah"
while True:
  r = requests.get(endpoint_url_fmt, params = params)

For some reason the go code is not reusing http connections.

EDIT : The go code needs the body to be closed to reuse the connection.

 resp , err = transport.RoundTrip(outboundRequest)
 resp.Close() //  This allows the connection to be reused

Answer 1

Based on further clarification from the OP. The default client does reuse connections.

Be sure to close the response.

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Additionally, I've found that I also needed to read until the response was complete before calling Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

To ensure http.Client connection reuse be sure to do two things:

• Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
• Call Body.Close()

---

Old answer, useful for rate limiting, but not what the OP was after:

I don't think setting max connections is possible via the golang 1.1 http APIs. This means you can shoot yourself in the foot with tons of TCP connections (until you run out of file descriptors or whatever) if you aren't careful.

That said, you could limit the rate at which you call the go routine for a particular host (and therefore outbound requests and connections) via time.Tick.

For example:

import "time"

requests_per_second := 5
throttle := time.Tick(1000000000 / requests_per_second)

for i := 0; i < 16; i += 1 {
  <-throttle  
  go serveQueue()
}