python urllib2 utf-8 encoding

Question

okay, I have: # -*- coding: utf-8 -*- in my python file.

the snippet:

opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
opener.addheaders = [('Accept-Charset', 'utf-8')]
f =opener.open(url)
doc = f.read().decode('utf-8')

The server response is: (via f.info())

Content-Type: text/html; charset=UTF-8

but i get the error:

UnicodeDecodeError: 'utf8' codec can't decode byte[...]: invalid continuation byte

What's wrong here?

Answer 1

Try decoding the data using 'latin-1' to see what it looks like. What you're seeing indicates a UTF-8 decode error (see UnicodeDecodeError, invalid continuation byte ).

It would be helpful if you posted the result of list(f.read())[:100] so we can see the data.

FYI, putting # -*- coding: utf-8 -*- is unrelated to your issue. That encoding refers to the encoding of your python script itself, not the data it is handling :-)

Answer 2

That particular error is commonly caused by trying to decode using utf-8 when the string was actually encoded with latin1. See UnicodeDecodeError, invalid continuation byte for some more info.

I suspect that despite the header, the server is not returning utf8 encoded content.

A solution that might be worth pursuing is to use chardet to 'guess' which encoding is used. Despite chardet's awesomeness consider it a last resort however.