Ruby Net::HTTP not decoding gzip?

Question

I have ruby-1.9.3-p327 with zlib installed. localhost:80 is the nginx simple test page.

require "net/http"
=> true
Net::HTTP::HAVE_ZLIB
=> true

res = Net::HTTP.start("localhost", "80") do |http|
  req = Net::HTTP::Get.new "/"
  req["accept-encoding"] = "gzip"
  http.request req
end
=> #<Net::HTTPOK 200 OK readbody=true>

res.get_fields "content-encoding"
=> ["gzip"]
res.body
=> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03\xEC\xBDi..."

The body was not decoded. Why?

Answer 1

For anyone, who get problem with code worked on ruby 1.9 and failed to work on upgrade to ruby 2.0, just include that code to your project.

module HTTPResponseDecodeContentOverride
  def initialize(h,c,m)
    super(h,c,m)
    @decode_content = true
  end
  def body
    res = super
    if self['content-length']
      self['content-length']= res.bytesize
    end
    res
  end
end
module Net
  class HTTPResponse
    prepend HTTPResponseDecodeContentOverride
  end
end

Answer 2

If you use http.get it should decode it automaticlly, but it looks like request might not do it for you.

There clearly is code to decompress the gzip request here, but only for the get method: https://github.com/ruby/ruby/blob/v1_9_3_327/lib/net/http.rb#L1031

Answer 3

I think it does not do it automatically.

To decode, try the following snippet (assuming the response is a StringIO):

begin
  Zlib::GzipReader.new(response).read
rescue Zlib::GzipFile::Error, Zlib::Error # Not gzipped
  response.rewind
  response.read
end

Answer 4

Based on my experiments, at least one reason this happens is because of the right_http_connection gem. I tested versions 1.3.0 and 1.4.0. This gem monkey patches Net::HTTP and causes problems with decoding GZipped responses.

You can read more about this problem in this GitHub issue.