Django + sqlite + Unicode

Question

I faced a problem with Unicode strings while adding new records to a sqlite database through the admin site.

class Translation(BaseModel):
  .....
  translation = models.CharField(max_length=100)

When I try to insert a word like 'été' an error occurs:

**UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)**

Update: Added traceback: http://pastebin.com/yLYFNDGB

Answer 1

When you assign é as a string to a variable, it actually takes it as xe9.

Like if you go to your python command line editor and write following code:

>>> a = u'café'
>>> a 

and press enter, the é character is replaced by \xe9 as illustrated as follows:

>>> u'caf\xe9'

on the other hand, if you write this with print statement, you would get what you initially assigned, i.e:

>>> print a
café

So, after making this point, the solution to your problem is using utf8 encoding to get rid of \xe9. So you can encode your string in the following way:

>>> string_for_output = yourstring-with-é.encode('utf8', 'replace')

For more about this topic, you may consult THIS URL . It discusses your problem under the heading "Frustration #3: Inconsistent treatment of output"

Hope this helps.

Answer 2

I found a solution. Actually, the problem was not in Django or sqlite. The issue was with the unicode() method.

Previously it was:

def __unicode__(self):
    return "{} ({})".format(self.translation, self.word.lang.abbr)

After an obvious fix, the problem is gone:

def __unicode__(self):
    return u"{} ({})".format(self.translation, self.word.lang.abbr)