twitter trends api UnicodeDecodeError: 'utf8&#

Question

I am trying to follow the sample code of the book "Mining the social web", 1-3.

I know its old so I follow the new sample from the web page enter link description here

BUT, SOMETIMES, I will suffer a Error info when I implement the code:

[ trend.decode('utf-8') for trend in world_trends()[0]['trends'] ]

And the error info is like this:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/twitter/api.py", line 167, in __call__
File "build/bdist.macosx-10.6-universal/egg/twitter/api.py", line 173, in _handle_response
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte

It doesnt always happen, but I think no programmer likes such a "random" case.

So could anyone help me on this issue? Whats the problem and how I can solve this?

Great thanks~

Answer 1

By default decode() will throw an error if it encounters a byte that it doesn't know how to decode.

You can use trend.decode('utf-8', 'replace') or trend.decode('utf-8', 'ignore') to not throw an error and silently ignore it.

Documentation on decode() here.

Answer 2

byte 0x8b in position 1 usually signals that the data stream is gzipped. For similar problems, see here and here.

To unzip the data stream:

buf = StringIO.StringIO(<response object>.content)
gzip_f = gzip.GzipFile(fileobj=buf)
content = gzip_f.read()