Escape unescaped characters in XML with Python

2020-02-10 06:57发布

站内问答 / Python
1582 4 4

I need to escape special characters in an invalid XML file which is about 5000 lines long. Here's an example of the XML that I have to deal with:

<root>
 <element>
  <name>name & surname</name>
  <mail>name@name.org</mail>
 </element>
</root>

Here the problem is the character "&" in the name. How would you escape special characters like this with a Python library? I didn't find a way to do it with BeautifulSoup.

4条回答
叛逆
2楼-- · 2020-02-10 07:24

If you don't care about invalid characters in the xml you could use XML parser's recover option (see Parsing broken XML with lxml.etree.iterparse):

from lxml import etree

parser = etree.XMLParser(recover=True) # recover from bad characters.
root = etree.fromstring(broken_xml, parser=parser)
print etree.tostring(root)

Output

<root>
<element>
<name>name  surname</name>
<mail>name@name.org</mail>
</element>
</root>
查看更多
0人赞 添加讨论(0) 举报
Evening l夕情丶
3楼-- · 2020-02-10 07:24

You're probably just wanting to do some simple regexp-ery on the HTML before throwing it into BeautifulSoup.

Even simpler, if there aren't any SGML entities (&...;) in the code, html=html.replace('&','&amp;') will do the trick.

Otherwise, try this:

x ="<html><h1>Fish & Chips & Gravy</h1><p>Fish &amp; Chips &#x0026; Gravy</p>"
import re
q=re.sub(r'&([^a-zA-Z#])',r'&amp;\1',x)
print q

Essentially the regex looks for & not followed by alpha-numeric or # characters. It won't deal with ampersands at the end of lines, but that's probably fixable.

查看更多
0人赞 添加讨论(0) 举报
爷的心禁止访问
4楼-- · 2020-02-10 07:26

This answer provides XML sanitizer functions, although they don't escape the unescaped characters, but simply drop them instead.

Using bs4 with lxml

The question wondered how to do it with Beautiful Soup. Here is a function which will sanitize a small XML bytes object with it. It was tested with the package requirements beautifulsoup4==4.8.0 and lxml==4.4.0. Note that lxml is required here by bs4.

import xml.etree.ElementTree

import bs4


def sanitize_xml(content: bytes) -> bytes:
    # Ref: https://stackoverflow.com/a/57450722/
    try:
        xml.etree.ElementTree.fromstring(content)
    except xml.etree.ElementTree.ParseError:
        return bs4.BeautifulSoup(content, features='lxml-xml').encode()
    return content  # already valid XML

Using only lxml

Obviously there is not much of a point in using both bs4 and lxml when this can be done with lxml alone. This lxml==4.4.0 using sanitizer function is essentially derived from the answer by jfs.

import lxml.etree


def sanitize_xml(content: bytes) -> bytes:
    # Ref: https://stackoverflow.com/a/57450722/
    try:
        lxml.etree.fromstring(content)
    except lxml.etree.XMLSyntaxError:
        root = lxml.etree.fromstring(content, parser=lxml.etree.XMLParser(recover=True))
        return lxml.etree.tostring(root)
    return content  # already valid XML
查看更多
0人赞 添加讨论(0) 举报
我只想做你的唯一
5楼-- · 2020-02-10 07:29 
<name>name & surname</name>

is not well-formed XML. It should be:

<name>name &amp; surname</name>

All conformant XML tools should create this - you normally do not have to worry. If you create a string with the '&' character then an XML tool will output the escaped version. If you create the string by hand it is your responsibility to make sure it is escaped. If you use an XML editor it should escape it for you.

If the file has been given you by someone else, send it back and tell them it is not well-formed. If they no longer exist you will have to use a plain text editor. That's fragile and messy but there is no other way. If the file has ampersands elsewhere that are used for escaping then the file is garbage.

See a 10-year-old post here and a later one here.

查看更多
0人赞 添加讨论(0) 举报
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(4)