Parse XHTML5 with undefined entities

Please consider this:

import xml.etree.ElementTree as ET

xhtml = '''<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''
parser = ET.XMLParser()
parser.entity['nbsp'] = '&#x00A0;'
tree = ET.fromstring(xhtml, parser=parser)
print(ET.tostring(tree, method='xml'))

which renders nice text representation of xhtml string.

But, for same XHTML document with HTML5 doctype:

xhtml = '''<!DOCTYPE html>
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''

I get Exception:

xml.etree.ElementTree.ParseError: undefined entity: line 5, column 19

so the parser can't handle it, although I added nbsp to entities dict.

Same happens if I use lxml:

from lxml import etree
parser = etree.XMLParser(resolve_entities=False)
tree = etree.fromstring(xhtml, parser=parser)
print etree.tostring(tree, method='xml')

raises:

lxml.etree.XMLSyntaxError: Entity 'nbsp' not defined, line 5, column 26

although I've set the parser to ignore entities.

Why is this, and how to make parsing of XHTML files with HTML5 doctype declaration possible?

Partial solution for lxml is to use recoverer:

parser = etree.XMLParser(resolve_entities=False, recover=True)

but I'm still waiting for better one.

The problem here is, the Expat parser used behind the scenes won't usually report unknown entities - it will rather throw an error, so the fallback code in xml.etree.ElementTree you were trying to trigger won't even run. You can use the UseForeignDTD method to change this behavior, it will make Expat ignore the doctype declaration and pass all entity declarations to xml.etree.ElementTree. The following code works correctly:

import xml.etree.ElementTree as ET

xhtml = '''<!DOCTYPE html>
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''
parser = ET.XMLParser()
parser._parser.UseForeignDTD(True)
parser.entity['nbsp'] = u'\u00A0'
tree = ET.fromstring(xhtml, parser=parser)
print(ET.tostring(tree, method='xml'))

The side-effect of this approach: as I said, the doctype declaration is completely ignored. This means that you have to declare all entities, even the ones supposedly covered by the doctype.

Note that the values you put into ElementTree.XMLParser.entity dictionary have to be regular strings, text that the entity will be replaced by - you can no longer refer to other entities there. So it should be u'\u00A0' for  .