saving an 'lxml.etree._ElementTree' object

Question

I've spent the last couple of days getting to grips with the basics of lxml; in particular using lxml.html to parse websites and create an ElementTree of the content. Ideally, I want to save the returned ElementTree so that I can load it up and experiment with it, without having to parse the website every time I modify my script. I assumed that pickling would be the way to go, however I'm now beginning to wonder. Although I am able to retrieve an ElementTree object after pickling...

type(myObject) 

returns

<class 'lxml.etree._ElementTree'>

the object itself appears to be 'empty', since none of the subsequent method/attribute calls I make on it yield any output.

My guess is that pickling isn't appropriate here, but can anyone suggest an alternative?

(In case it matters, the above is happening in: python3.2, lxml 2.3.2, snow-leopard))

Answer 1

lxml is a C library - libxml to be precise - and the object probably don't support python pickling or any other kind of serialization - except serializing them to XML.

So you'll either have to keep them in memory, or re-parse the XML fragments you need, I assume.

Answer 2

You are already dealing with XML, and lxml is great at parsing XML. So I think the simplest thing to do would be to serialize to XML:

To write to file:

import lxml.etree as ET

filename = '/tmp/test.xml'
myobject.write(filename)

To call the write method, note that myobject must be an lxml.etree._ElementTree. If it is an lxml.etree._Element, then you would need myobject.getroottree().write(filename).

To parse from file name/path, file object, or URL:

myobject = ET.parse(file_or_url)

To parse from string:

myobject = ET.fromstring(content)

Answer 3

I don't believe you can pickle lxml instances, but what I did because I was in a similar situation was I pickled the object instances that would build the tree.

Each instance and its child had a function to build the Element tree. So I would simply pickle/cache the Python object, fetch it from cache, and then call the build functions to get my Element tree.