How to get the html source of a specific element w

Question

The page I'm looking at contains :

<div id='1'> <p> text 1 <h1> text 2 </h1> text 3 <p> text 4 </p> </p> </div>

I want to get all the text in the div, except for the text that is in the <h>. (I want to get "text 1","text 3" and "text 4") There may be a few <h> elements, or none at all. And there may be a few <p> elements, even one inside the other, or none.

I thought to do this by getting all the html source of the div, and using a regex to remove the <h> elements. But selenium.get_text does not return the html, just the text (all of it!).

I know I can use selenium.get_html_source and then look for the element I need with a regex, but that looks like a waste since selenium knows how to find the element.

Does anyone have a better solution? Thanks :)

Answer 1

What about using jQuery?

Edit:

First you have to add the required .JS files, for that go to www.jQuery.com.

Then all you need to do is call a simple jQuery selector:

alert($("div#1").html());

Answer 2

Use xpath. From selenium.py:

Without an explicit locator prefix, Selenium uses the following default strategies:

• \**dom**\ , for locators starting with "document."
• \**xpath**\ , for locators starting with "//"
• \**identifier**\ , otherwise

In your case, you could try

selenium.get_text("//div[@id='1']/descendant::*[not(self::h1)]")

You can learn more about xpath here.

P.S. I don't know if there's good HTML documentation available for python-selenium, but I haven't found any; on the other hand, the docstrings of the selenium.py file seem to constitute comprehensive documentation. So I'd suggest looking up the source to get a better understanding of how it works.

Answer 3

The following code will give you the HTML in the div element:

sel = selenium('localhost', 4444, browser, my_url)
html = sel.get_eval("this.browserbot.getCurrentWindow().document.getElementById('1').innerHTML")

then you can use BeautifulSoup to parse it and extract what you really want.

I hope it helps

Answer 4

The selected answer does not work in Python 3 at the time of writing. Instead use this:

from selenium import webdriver

wd = webdriver.Firefox()
wd.get(url)
return wd.execute_script('return window.document.getElementById('1').innerHTML')