XPath axis, get all following nodes until

Question

I have the following example of HTML:

<!-- lots of html -->
<h2>Foo bar</h2>
<p>lorem</p>
<p>ipsum</p>
<p>etc</p>

<h2>Bar baz</h2>
<p>dum dum dum</p>
<p>poopfiddles</p>
<!-- lots more html ... -->

I'm looking to extract all paragraphs following the 'Foo bar' header, until I reach the 'Bar baz' header (the text for the 'Bar baz' header is unknown, so unfortunately I can't use the answer provided by bougyman). Now I can of course using something like //h2[text()='Foo bar']/following::p but that of course will grab all paragraphs following this header. So I have the option to traverse the nodeset and push paragraphs into an Array until the text matches that of the next following header, but let's be honest, that's never as cool as being able to do it in XPath.

Is there a way to do this that I'm missing?

Answer 1

In XPath 2.0 (I know this doesn't help you...) the simplest solution is probably

h2[. = 'Foo bar']/following-sibling::* except h2[. = 'Bar baz']/(.|following-sibling::* )

But like other solutions presented, this is likely (in the absence of an optimizer that recognizes the pattern) to be linear in the number of elements beyond the second h2, whereas you would really like a solution whose performance depends only on the number of elements selected. I've always felt it would be nice to have an until operator:

h2[. = 'Foo bar']/(following-sibling::* until . = 'Bar baz')

In its absence an XSLT or XQuery solution using recursion is likely to perform better when the number of nodes to be selected is small compared with the number of following siblings.

Answer 2

XPath 2.0 has the operator << (with $node1 << $node2 being true if $node1 precedes $node2) so that way you can use //h2[. = 'Foo bar']/following-sibling::p[. << //h2[. = 'Bar baz']]. I don't know however what nokogiri is respectively whether it supports XPath 2.0.

Answer 3

Just because it's not between the answers, the classic XPath 1.0 set exclusion:

A - B = $A[count(.|$B)!=count($B)]

For this case:

(//h2[.='Foo bar']
    /following-sibling::p)
       [count(.|../h2[.='Foo bar']
                     /following-sibling::h2[1]
                        /following-sibling::p)
        != count(../h2[.='Foo bar']
                     /following-sibling::h2[1]
                        /following-sibling::p)]

Note: This would be the negation of Kaysian Method.

Answer 4

require 'nokogiri'

doc = Nokogiri::XML <<ENDXML
<root>
  <h2>Foo</h2>
  <p>lorem</p>
  <p>ipsum</p>
  <p>etc</p>

  <h2>Bar</h2>
  <p>dum dum dum</p>
  <p>poopfiddles</p>
</root>
ENDXML

a = doc.xpath( '//h2[text()="Foo"]/following::p[not(preceding::h2[text()="Bar"])]' )
puts a.map{ |n| n.to_s }
#=> <p>lorem</p>
#=> <p>ipsum</p>
#=> <p>etc</p>

I suspected that it might be more efficient to just walk the DOM using next_sibling until you hit the end:

node = doc.at_xpath('//h2[text()="Foo bar"]').next_sibling
stop = doc.at_xpath('//h2[text()="Bar baz"]')
a = []
while node && node!=stop
  a << node unless node.type == 3 # skip text nodes
  node = node.next_sibling
end

puts a.map{ |n| n.to_s }
#=> <p>lorem</p>
#=> <p>ipsum</p>
#=> <p>etc</p>

However, this is NOT faster. In a few simple tests, I found that xpath-only (the first solution) is about 2x as fast as this looping test, even when there are a very large number of paragraphs after the stop node. When there are many nodes to capture (and few after the stop) it performs even better, in the 6x-10x range.

Answer 5

how about matching on the second one? If you only want the top section, match the second and grab everything above it .
doc.xpath("//h2[text()='Bar baz']/preceding-sibling::p").map { |m| m.text } => ["lorem", "ipsum", "etc"]

or if you don't know the second one, go another level with: doc.xpath("//h2[text()='Foo bar']/following-sibling::h2/preceding-sibling::p").map { |it| it.text } => ["lorem", "ipsum", "etc"]

Answer 6

This XPATH 1.0 statement selects all of the <p> that are siblings that follow an <h2> who's string value is equal to "Foo bar", that are also followed by an <h2> sibling element who's first preceding sibling <h2> has a string value of "Foo bar".

//p[preceding-sibling::h2[.='Foo bar']]
 [following-sibling::h2[
  preceding-sibling::h2[1][.='Foo bar']]]