Convert XML/HTML Entities into Unicode String in P

2018-12-31 09:31发布

站内问答 / Python
1420 10 4

This question already has an answer here:

I'm doing some web scraping and sites frequently use HTML entities to represent non ascii characters. Does Python have a utility that takes a string with HTML entities and returns a unicode type?

For example:

I get back:

ǎ

which represents an "ǎ" with a tone mark. In binary, this is represented as the 16 bit 01ce. I want to convert the html entity into the value u'\u01ce'

10条回答
浅入江南
2楼-- · 2018-12-31 09:49

This is a function which should help you to get it right and convert entities back to utf-8 characters.

def unescape(text):
   """Removes HTML or XML character references 
      and entities from a text string.
   @param text The HTML (or XML) source text.
   @return The plain text, as a Unicode string, if necessary.
   from Fredrik Lundh
   2008-01-03: input only unicode characters string.
   http://effbot.org/zone/re-sub.htm#unescape-html
   """
   def fixup(m):
      text = m.group(0)
      if text[:2] == "&#":
         # character reference
         try:
            if text[:3] == "&#x":
               return unichr(int(text[3:-1], 16))
            else:
               return unichr(int(text[2:-1]))
         except ValueError:
            print "Value Error"
            pass
      else:
         # named entity
         # reescape the reserved characters.
         try:
            if text[1:-1] == "amp":
               text = "&"
            elif text[1:-1] == "gt":
               text = ">"
            elif text[1:-1] == "lt":
               text = "<"
            else:
               print text[1:-1]
               text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
         except KeyError:
            print "keyerror"
            pass
      return text # leave as is
   return re.sub("&#?\w+;", fixup, text)
查看更多
0人赞 添加讨论(0) 举报
只靠听说
3楼-- · 2018-12-31 09:55

The standard lib’s very own HTMLParser has an undocumented function unescape() which does exactly what you think it does:

import HTMLParser
h = HTMLParser.HTMLParser()
h.unescape('© 2010') # u'\xa9 2010'
h.unescape('© 2010') # u'\xa9 2010'
查看更多
0人赞 添加讨论(0) 举报
骚的不知所云
4楼-- · 2018-12-31 09:55

Use the builtin unichr -- BeautifulSoup isn't necessary:

>>> entity = '&#x01ce'
>>> unichr(int(entity[3:],16))
u'\u01ce'
查看更多
0人赞 添加讨论(0) 举报
牵手、夕阳
5楼-- · 2018-12-31 09:58

You could find an answer here -- Getting international characters from a web page?

EDIT: It seems like BeautifulSoup doesn't convert entities written in hexadecimal form. It can be fixed:

import copy, re
from BeautifulSoup import BeautifulSoup

hexentityMassage = copy.copy(BeautifulSoup.MARKUP_MASSAGE)
# replace hexadecimal character reference by decimal one
hexentityMassage += [(re.compile('&#x([^;]+);'), 
                     lambda m: '&#%d;' % int(m.group(1), 16))]

def convert(html):
    return BeautifulSoup(html,
        convertEntities=BeautifulSoup.HTML_ENTITIES,
        markupMassage=hexentityMassage).contents[0].string

html = '<html>&#x01ce;&#462;</html>'
print repr(convert(html))
# u'\u01ce\u01ce'

EDIT:

unescape() function mentioned by @dF which uses htmlentitydefs standard module and unichr() might be more appropriate in this case.

查看更多
0人赞 添加讨论(0) 举报
上一页 1 2
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(4)