What is a good XML stream parser for Python? [clos

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 4 months ago.

Are there any XML parsers for Python that can parse file streams? My XML files are too big to fit in memory, so I need to parse the stream.

Ideally I wouldn't have to have root access to install things, so lxml is not a very good option.

I have been using xml.etree.ElementTree but I am convinced it is broken.

Answer 1

Are you looking for xml.sax? It's right in the standard library.

Answer 2

Here's good answer about xml.etree.ElementTree.iterparse practice on huge XML files. lxml has the method as well. The key to stream parsing with iterparse is manual clearing and removing already processed nodes, because otherwise you will end up running out of memory.

Another option is using xml.sax. The official manual is too formal to me, and lacks examples so it needs clarification along with the question. Default parser module, xml.sax.expatreader, implement incremental parsing interface xml.sax.xmlreader.IncrementalParser. That is to say xml.sax.make_parser() provides suitable stream parser.

For instance, given a XML stream like:

<?xml version="1.0" encoding="utf-8"?>
<root>
  <entry><a>value 0</a><b foo='bar' /></entry>
  <entry><a>value 1</a><b foo='baz' /></entry>
  <entry><a>value 2</a><b foo='quz' /></entry>
  ...
</root>

Can be handled in the following way.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import xml.sax


class StreamHandler(xml.sax.handler.ContentHandler):

  lastEntry = None
  lastName  = None


  def startElement(self, name, attrs):
    self.lastName = name
    if name == 'entry':
      self.lastEntry = {}
    elif name != 'root':
      self.lastEntry[name] = {'attrs': attrs, 'content': ''}

  def endElement(self, name):
    if name == 'entry':
      print({
        'a' : self.lastEntry['a']['content'],
        'b' : self.lastEntry['b']['attrs'].getValue('foo')
      })
      self.lastEntry = None
    elif name == 'root':
      raise StopIteration

  def characters(self, content):
    if self.lastEntry:
      self.lastEntry[self.lastName]['content'] += content


if __name__ == '__main__':
  # use default ``xml.sax.expatreader``
  parser = xml.sax.make_parser()
  parser.setContentHandler(StreamHandler())
  # feed the parser with small chunks to simulate
  with open('data.xml') as f:
    while True:
      buffer = f.read(16)
      if buffer:
        try:
          parser.feed(buffer)
        except StopIteration:
          break
  # if you can provide a file-like object it's as simple as
  with open('data.xml') as f:
    parser.parse(f)

Answer 3

Use xml.etree.cElementTree. It's much faster than xml.etree.ElementTree. Neither of them are broken. Your files are broken (see my answer to your other question).