Normalizing Unicode

Question

Is there a standard way, in Python, to normalize a unicode string, so that it only comprehends the simplest unicode entities that can be used to represent it ?

I mean, something which would translate a sequence like ['LATIN SMALL LETTER A', 'COMBINING ACUTE ACCENT'] to ['LATIN SMALL LETTER A WITH ACUTE'] ?

See where is the problem:

>>> import unicodedata
>>> char = "á"
>>> len(char)
1
>>> [ unicodedata.name(c) for c in char ]
['LATIN SMALL LETTER A WITH ACUTE']

But now:

>>> char = "á"
>>> len(char)
2
>>> [ unicodedata.name(c) for c in char ]
['LATIN SMALL LETTER A', 'COMBINING ACUTE ACCENT']

I could, of course, iterate over all the chars and do manual replacements, etc., but it is not efficient, and I'm pretty sure I would miss half of the special cases, and do mistakes.

Answer 1

Yes, there is.

unicodedata.normalize(form, unistr)

You need to select one of the four normalization forms.

Answer 2

The unicodedata module offers a .normalize() function, you want to normalize to the NFC form:

>>> unicodedata.normalize('NFC', u'\u0061\u0301')
u'\xe1'
>>> unicodedata.normalize('NFD', u'\u00e1')
u'a\u0301'

NFC, or 'Normal Form Composed' returns composed characters, NFD, 'Normal Form Decomposed' gives you decomposed, combined characters.

The additional NFKC and NFKD forms deal with compatibility codepoints; e.g. U+2160 (ROMAN NUMERAL ONE) is really just the same thing as U+0049 (LATIN CAPITAL LETTER I) but present in the Unicode standard to remain compatible with encodings that treat them separately. Using either NFKC or NFKD form, in addition to composing or decomposing characters, will also replace all 'compatibility' characters with their canonical form:

>>> unicodedata.normalize('NFC', u'\u2167')  # roman numeral VIII
u'\u2167'
>>> unicodedata.normalize('NFKC', u'\u2167') # roman numeral VIII
u'VIII'

Note that there is no guarantee that composed and decomposed forms are communicative; normalizing a combined character to NFC form, then converting the result back to NFD form does not always result in the same character sequence. The Unicode standard maintains a list of exceptions; characters on this list are composable, but not decomposable back to their combined form, for various reasons. Also see the documentation on the Composition Exclusion Table.