PG::Error: ERROR: invalid byte sequence for encodi

Question

I'm getting the following ActiveRecord error when certain strings are saved to the database.

ActiveRecord::StatementInvalid: PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0xfc

I think it's happening for this string Mühldorf.

I've tried adding # encoding: utf-8 to the top of my ruby files to solve this, but doesn't seem to be doing anything.

I'm pulling location data using Ruby Geocoder, and that's where the string is coming from.

I'm running Ruby 1.9.3 on Postgres database on Heroku.

Answer 1

A lower case U-umlaut is 0xfc in ISO 8859-1 (AKA Latin-1) but 0xfc is not a valid UTF-8 character. The problem is that you have a Latin-1 string that you're trying to treat as UTF-8 and PostgreSQL is rightly complaining.

Either fix the data source to send you UTF-8 or, if it will always send you Latin-1, fix the encoding yourself with something like:

utf_8_string = latin_1_string.force_encoding('iso8859-1').encode('utf-8')

and then work with the utf_8_string version.