POST binary data using httplib cause Unicode excep

Question

When i try to send an image with urllib2 the UnicodeDecodeError exception is occured.

HTTP Post body:

f = open(imagepath, "rb")
binary = f.read()
mimetype, devnull = mimetypes.guess_type(urllib.pathname2url(imagepath))

body = """Content-Length: {size}
Content-Type: {mimetype}

{binary}
""".format(size=os.path.getsize(imagepath),  
           mimetype=mimetype,
           binary=binary)

request = urllib2.Request(url, body, headers)
opener = urllib2.build_opener(urllib2.HTTPSHandler(debuglevel=1))
response = opener.open(request)
print response.read()

Traceback :

   response = opener.open(request)
  File "/usr/local/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 827, in _send_output
    msg += message_body
  File "/home/usertmp/biogeek/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 49: invalid start byte

python version 2.7.5

Anyone know a solution to this?

Answer 1

You're trying to send a body containing headers and content. If you want to send content type and content length, you need to do it in the headers, not in the body:

headers = {'Content-Type': mimetype, 'Content-Length', str(size)}
request = urllib2.Request(url, data=binary, headers=headers)

If you don't set the Content-Length header, it will be automatically set to the size of data

As to your error: it's happening on the line

msg += message_body

This error can only happen, if one of these two strings is unicode, and the other str containing \xff, as in that case the latter will be automatically coecred to unicode using sys.getdefaultencoding().

My final guess would be: message_body here is your data, which is a str and contains \xff somewhere. msg is what has been passed to the HTTPConnection earlier, namely the headers, and they are unicode because you either used unicode for at least one key in your headers (the values are converted to str earlier), or you have imported unicode_literals from __futures__.