BeautifulSoup4 stripped_strings gives me byte obje

问题:

I'm trying to get the text out of a blockquote which looks like this:

<blockquote class="postcontent restore ">
    01 Oyasumi
    <br></br>
    02 DanSin'
    <br></br>
    03 w.t.s.
    <br></br>
    04 Lovism
    <br></br>
    05 NoName
    <br></br>
    06 Gakkou
    <br></br>
    07 Happy☆Day
    <br></br>
    08 Endless End.
</blockquote>

What I'm trying to do is this in python 2.7 (it can't decode the ☆ character which is why I tried to use encode):

soup = BeautifulSoup(r.text, "html5lib") #r is from a requests get request
content = soup.find("blockquote", {"class": "postcontent restore "}).stripped_strings
for line in content:
    print(line.encode("utf-8"))

And this is what I get:

b'01 Oyasumi'
b"02 DanSin'"
b'03 w.t.s.'
b'04 Lovism'
b'05 NoName'
b'06 Gakkou'
b'07 Happy\xe2\x98\x86Day'
b'08 Endless End.'

What am I doing wrong?

回答1:

The problem is that Beautiful Soup converts the original encoding to Unicode if the from_encoding is not specified using a sub-library called Unicode, Dammit. More info in the Encodings section in the documentation.

>>> from bs4 import BeautifulSoup
>>> doc = '''<blockquote class="postcontent restore ">
...     01 Oyasumi
...     <br></br>
...     02 DanSin'
...     <br></br>
...     03 w.t.s.
...     <br></br>
...     04 Lovism
...     <br></br>
...     05 NoName
...     <br></br>
...     06 Gakkou
...     <br></br>
...     07 Happy☆Day
...     <br></br>
...     08 Endless End.
... </blockquote>'''
>>> soup = BeautifulSoup(doc, 'html5lib')
>>> soup.original_encoding 
u'windows-1252'
>>> content = soup.find("blockquote", {"class": "postcontent restore "}).stripped_strings
>>> for line in content:
...     print(line)
... 
01 Oyasumi
02 DanSin'
03 w.t.s.
04 Lovism
05 NoName
06 Gakkou
07 Happy☆Day
08 Endless End.

To fix this you have two options:

1. By passing in the correct from_encoding parameter or excluding the wrong the wrong encoding Dammit is guessing. One problem is that not all Parsers support the exclude_encodings argument. For example the html5lib tree builder doesn't support exclude_encoding

   >>> soup = BeautifulSoup(doc, 'html5lib', from_encoding='utf-8')
   >>> content = soup.find("blockquote", {"class": "postcontent restore "}).stripped_strings
   >>> for line in content:
   ...     print(line)
   ... 
   01 Oyasumi
   02 DanSin'
   03 w.t.s.
   04 Lovism
   05 NoName
   06 Gakkou
   07 Happy☆Day
   08 Endless End.
   >>> 
   

2. Use the lxml Parser

   >>> soup = BS(doc, 'lxml')
   >>> soup.original_encoding
   'utf-8'
   >>> content = soup.find("blockquote", {"class": "postcontent restore "}).stripped_strings
   >>> for line in content:
   ...     print(line)
   ... 
   01 Oyasumi
   02 DanSin'
   03 w.t.s.
   04 Lovism
   05 NoName
   06 Gakkou
   07 Happy☆Day
   08 Endless End.