Specify Character Encoding for Net::HTTP

Question

When I make this HTTP request:

Net::HTTP.get_response('www.telize.com',"/geoip/190.88.39.27").body
  => "{\"timezone\":\"America\\/Curacao\",\"isp\":\"United Telecommunication Services (UTS)\",\"country\":\"Cura\xE7ao\",\"dma_code\":\"0\",\"region_code\":\"00\",\"area_code\":\"0\",\"ip\":\"190.88.39.27\",\"asn\":\"AS11081\",\"continent_code\":\"NA\",\"city\":\"Willemstad\",\"longitude\":-68.9167,\"latitude\":12.1,\"country_code\":\"CW\",\"country_code3\":\"CUW\"}\n"

It returns a JSON body, but notice the country: \"country\":\"Cura\xE7ao\". The response body should actually looks like this: "country":"Curaçao". It looks like Net::HTTP is assuming this is ASCII-8BIT:

Net::HTTP.get_response('www.telize.com',"/geoip/190.88.39.27").body.encoding
 => Encoding:ASCII-8BIT

but this can't be the case. How can I tell Net::HTTP which character encoding to use when making the request?

Answer 1

You can't tell a server what encoding to use, but you can ask it what it thinks the file's encoding is and then pass that Net::HTTP.

Look at the head method:

response = nil
Net::HTTP.start('www.telize.com',80) { |http|
  response = http.head('/geoip/190.88.39.27')
}
response.each_header { |h| p "#{ h } => #{ response[h] }" }

Running that tells you the contents of the various headers:

"server => nginx"
"date => Thu, 12 Jun 2014 23:42:16 GMT"
"content-type => application/json; charset=iso-8859-1"
"connection => close"

The content-type value is what you want:

response['content-type'].split('=').last
# => "iso-8859-1"

Note that the server rarely does a consistency check to see whether the encoding it's told to use actually matches the file it's serving. This means the content you receive could vary wildly from what the server said it is, and, at that point, you're totally on your own to figure out what it really is, especially when the file has mixed encodings. Welcome to the wild and wooly internet.

Answer 2

As the Tin Man determined, "\xE7" is the latin-1 encoding for LATIN SMALL LETTER C WITH CEDILLA, which as far as I can determine isn't a valid json encoding.

But...once you know the encoding, you can change it from ruby's ASCII-8BIT(which just means ruby considers the data to be binary, i.e. unencoded) to UTF-8, like this:

require 'net/http'

server_encoding = "ISO-8859-1"
resp = Net::HTTP.get_response('www.telize.com',"/geoip/190.88.39.27")
json = resp.body.force_encoding(server_encoding).encode("UTF-8")
puts json

--output:--

{"timezone":"America\/Curacao","isp":"United Telecommunication Services
UTS)","country":"Curaçao","dma_code":"0","region_code":"00","area_code":"0",
"ip":"190.88.39.27","asn":"AS11081","continent_code":"NA","city":"Willemstad",
"longitude":-68.9167,"latitude":12.1,"country_code":"CW","country_code3":"CUW"}

It looks like Net::HTTP is assuming this is ASCII-8BIT

Net::HTTP tags the data as binary/ASCII-8BIT, i.e. the data has no encoding, and leaves it to you to figure out how to interpret the data.