Unicode search not working

Question

Consider this.

# -*- coding: utf-8 -*-
data = "cdbsb \xe2\x80\xa6 abc"
print data 
#prints cdbsb … abc
              ^
print re.findall(ur"[\u2026]", data )

Why can't re find this unicode character ? I have already checked

\xe2\x80\xa6 === … === U+2026

Answer 1

An alternative solution:

>>> data = "cdbsb \xe2\x80\xa6 abc"
>>> print data 
cdbsb … abc
>>> if u"\u2026".encode('utf8') in data: print True
... 
True
>>> if u"\u2026" in data.decode('utf8'): print True
... 
True

Answer 2

data is of str type and contains ASCII character with hex value. But the search term is of unicode type . Print function converts default to sys.stdout.encoding. When I try to print data as it is, the output differs from data.decode('utf-8'). I am using Python 2.7

data = "cdbsb \xe2\x80\xa6 abc"
search = ur"[\u2026]"

print sys.stdout.encoding
## windows-1254

print data, type(data)
## cdbsb … abc <type 'str'>

print data.decode(sys.stdout.encoding)
## cdbsb … abc

print data.decode('utf-8')
## cdbsb … abc

print search, type(search)
## […] <type 'unicode'>

print re.findall(search, data.decode('utf-8'))
## [u'\u2026']

Answer 3

If you go through the link provided by nhahtdh

Solving Unicode Problems in Python 2.7

You can see the original string was in bytes and we were searching for unicode. So it should never have worked.

encode(): Gets you from Unicode → bytes

decode(): Gets you from bytes → Unicode

Following these we can solve it in 2 ways.

# -*- coding: utf-8 -*-
data = "cdbsb \xe2\x80\xa6 abc".decode("utf-8")  #convert to unicode
print data
print re.findall(ur"[\u2026]", data )
print re.findall(ur"[\u2026]", data )[0].encode("utf-8")  #compare with unicode byte string and then reconvert to bytes for print

data1 = "cdbsb \xe2\x80\xa6 abc"  #let it remain bytes
print data1
print re.findall(r"\xe2\x80\xa6", data1 )[0] #search for bytes

Answer 4

My guess is that the issue is because data is a byte-string. You might have the console encoding as utf-8 , hence when printing the string, the console converts the string to utf-8 and then shows it (You can check this out at sys.stdout.encoding ). Hence you are getting the character - … .

But most probably re does not do this decoding for you.

If you convert data to utf-8 encoding, you would get the desired result when using re.findall. Example -

>>> data = "cdbsb \xe2\x80\xa6 abc"
>>> print re.findall(ur"[\u2026]", data.decode('utf-8') )
[u'\u2026']