Encoding issue when using Nokogiri replace

问题:

I have this code:

# encoding: utf-8
require 'nokogiri'

s = "<a href='/path/to/file'>Café Verona</a>".encode('UTF-8')
puts "Original string: #{s}"

@doc = Nokogiri::HTML::DocumentFragment.parse(s)

links = @doc.css('a')
only_text = 'Café Verona'.encode('UTF-8')
puts "Replacement text: #{only_text}"
links.first.replace(only_text)
puts @doc.to_html

However, the output is this:

Original string: <a href='/path/to/file'>Café Verona</a>
Replacement text: Café Verona
Café Verona

Why does the text in @doc end up with the wrong encoding?

I tried with and without encode('UTF-8') or using Document instead of DocumentFragment, but it's the same problem.

I'm using Nokogiri v1.5.6 with Ruby 1.9.3p194.

回答1:

Seems that if you pass a nokogiri text object it does the thing ;)

links.first.replace Nokogiri::XML::Text.new(only_text, @doc)

回答2:

I can't duplicate the problem, but I have two different things to try:

Instead of using:

s = "<a href='/path/to/file'>Café Verona</a>".encode('UTF-8')

Try:

s = "<a href='/path/to/file'>Café Verona</a>"

Your string is already UTF-8 encoded, because of your statement # encoding: utf-8. That's why you put that in the script, to tell Ruby the literal string is in UTF-8. It's possible that you're double-encoding it, though I don't think Ruby will -- it should silently ignore the second attempt because it's already UTF-8.

Another thing I wonder about is, output like:

Café Verona

is an indicator that the language/character-set encoding of your system and your terminal aren't right. Trying to output UTF-8 strings on a system set to something else can get mismatches in the terminal and/or browser. Windows systems are typically Win-1252, ISO-8859-1 or something similar, not UTF-8. On my Mac OS system I have these environment variables set:

LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8

"Open iso-8859-1 encoded html with nokogiri messes up accents" might be useful too.