UnicodeDecodeError when passing GET data in Python

Question

This feels like a really basic question, but I haven't been able to find an answer.

I would like to read data from an url, for example GET data from a querystring. I am using the webapp framework in Python. I tried the following code, but since I've a total beginner at Python/appengine, I've certainly done something wrong.

class MainPage(webapp.RequestHandler):
    def get(self):
        self.response.out.write(self.request.get('data'))

application = webapp.WSGIApplication([('/', MainPage),('/search', Search),('/next', Next)],debug=False)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()

When testing in my test environment, the URL http://localhost/?data=test just returns this error message below. Without the querystring, it just displays a blank page as expected.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd6 in position 40: ordinal not in range(128)

What am I doing wrong and what should I do instead?

Answer 1

You try to e.g. print an ASCII coded string actually containing data of a different charset. This can happen e.g. with Latin-1 encoded data. Try converting your input to unicode using

unicoded = unicode(non_unicode_string, source_encoding)

where source_encoding is something like 'cp1252', 'iso-8859-1' etc., and sending this to output.

Have a look at this HOWTO. For a list of encodings supported by Python, see this

Answer 2

Check out this blog post on how to do unicode right in Python. In a nutshell, you're trying to decode a byte string (implicitly) as ASCII, and it contains a byte that isn't valid in that codec. Your string is probably in UTF-8.