一些“UTF-8”编解码器不能解码字节(Some 'utf-8' codec can

我得到一些错误，当我下载使用网站wget

码：

import threading
import urllib.request
import os
import re
import time
import json

def wget(url):
                #self.url = url 
    data = os.popen('wget -qO- %s'% url).read()
    return data

print (wget("http://jamesholm.se/dj.php"))

错误：

Traceback (most recent call last):
  File "stand-alone-check-url.py", line 13, in <module>
    print (wget("http://jamesholm.se/dj.php"))
  File "stand-alone-check-url.py", line 10, in wget
    data = os.popen('wget -qO- %s'% url).read()
  File "/usr/local/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 13133: invalid start byte

如何克服这个问题？

Answer 1:

你不能解码任意字节序列为UTF-8编码的文本：

>>> b'\xa9'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte

该页面显示，它使用UTF-8，但服务器发送的实际数据是不是UTF-8。 它发生。

有bs4.UnicodeDammit ，让您处理不一致的编码数据：

import bs4 # $ pip install beautifulsoup4

print(bs4.UnicodeDammit.detwingle(b'S\x9aben  - Ostwind Rec').decode('utf-8'))
# -> Sšben  - Ostwind Rec

Answer 2:

取而代之的wget的使用requests Python模块。

>>> import requests
>>> data = requests.get("http://jamesholm.se/dj.php").text
>>> print(data)

文章来源: Some 'utf-8' codec can't decode byte