This problem began as the commonly-seen “incompatible character encodings: ASCII-8BIT and UTF-8” problem, but that is not what I'm asking. Rather, I discovered that this problem was happening because certain fields of my database are being tagged as ASCII-8BIT when they're retrieved, while most are correctly shown as UTF-8.
For example, in a table with columns country and nationality, where both columns in row 16 have identical values (copied-and-pasted), I get
irb(main):003:0> c = Country.find(16)
irb(main):004:0> puts "#{c.name}, #{c.name.encoding}, #{c.name.bytes.to_a}"
�land Islands, UTF-8, [195, 133, 108, 97, 110, 100, 32, 73, 115, 108, 97, 110, 100, 115]
irb(main):005:0> puts "#{c.nationality}, #{c.nationality.encoding}, #{c.nationality.bytes.to_a}"
�land Islands, ASCII-8BIT, [195, 133, 108, 97, 110, 100, 32, 73, 115, 108, 97, 110, 100, 115]
Likewise, a simple puts name gives �land Islands while for nationality it gives "\xC3\x85land Islands" -- same bytes, different presentation.
The encoding for a given column appears to be constant regardless of whether the string has non-ascii characters, so it is not simply a problem with the string. That is, all the values in nationality are interpreted as ascii and all those in name as UTF-8.
The problem is not limited to a single table, and I have not found any pattern to which columns are mis-recognized.
Here are the settings and environment:
- Rails 3.0.0 on Windows 7 64-bit
- Database adapter: mysql2 and mysql both show same behavior
- Database.yml includes
encoding: utf8 - application.rb includes
config.encoding = "utf-8" - MySQL database, table, and both columns are defined as utf8
- Both columns in MySQL are varchar, 255, allow null
- I can reproduce the problem with a fresh installation of Rails and nothing except the Country model defined, to access the database. I have not yet tried with a fresh, one-line database.
Anyone know what's going on here?
