How do I scrape HTML between two HTML comments usi

I have some HTML pages where the contents to be extracted are marked with HTML comments like below.

<html>
 .....
<!-- begin content -->
 <div>some text</div>
 <div><p>Some more elements</p></div>
<!-- end content -->
...
</html>

I am using Nokogiri and trying to extract the HTML between the  and  comments.

I want to extract the full elements between these two HTML comments:

<div>some text</div>
<div><p>Some more elements</p></div>

I can get the text-only version using this characters callback:

class TextExtractor < Nokogiri::XML::SAX::Document

  def initialize
    @interesting = false
    @text = ""
    @html = ""
  end

  def comment(string)
    case string.strip        # strip leading and trailing whitespaces
    when /^begin content/      # match starting comment
      @interesting = true
    when /^end content/
    @interesting = false   # match closing comment
  end

  def characters(string)
    @text << string if @interesting
  end

end

I get the text-only version with @text but I need the full HTML stored in @html.

标签： ruby-on-rails ruby web-scraping web-crawler nokogiri

1条回答

何必那么认真

2楼-- · 2019-05-26 05:52

Extracting content between two nodes is not a normal thing we'd do; Normally we'd want content inside a particular node. Comments are nodes, they're just special types of nodes.

require 'nokogiri'

doc = Nokogiri::HTML(<<EOT)
<body>
<!-- begin content -->
 <div>some text</div>
 <div><p>Some more elements</p></div>
<!-- end content -->
</body>
EOT

By looking for a comment containing the specified text it's possible to find a starting node:

start_comment = doc.at("//comment()[contains(.,'begin content')]") # => #<Nokogiri::XML::Comment:0x3fe94994268c " begin content ">

Once that's found then a loop is needed that stores the current node, then looks for the next sibling until it finds another comment:

content = Nokogiri::XML::NodeSet.new(doc)
contained_node = start_comment.next_sibling
loop do
  break if contained_node.comment?
  content << contained_node
  contained_node = contained_node.next_sibling
end

content.to_html # => "\n <div>some text</div>\n <div><p>Some more elements</p></div>\n"

0人赞添加讨论(0) 举报

How do I scrape HTML between two HTML comments usi

采纳回答

编辑标签

举报内容

检举类型

检举原因

检举说明(必填)

打开微信“扫一扫”，打开网页后点击屏幕右上角分享按钮

付费偷看金额在0.1-10元之间