extracting just page text using HTMLAgilityPack

Question

Ok so i am really new to XPath queries used in HTMLAgilityPack.

So lets consider this page http://health.yahoo.net/articles/healthcare/what-your-favorite-flavor-says-about-you. What i want is to extract just the page content and nothing else.

So for that i first remove script and style tags.

Document = new HtmlDocument();
        Document.LoadHtml(page);
        TempString = new StringBuilder();
        foreach (HtmlNode style in Document.DocumentNode.Descendants("style").ToArray())
        {
            style.Remove();
        }
        foreach (HtmlNode script in Document.DocumentNode.Descendants("script").ToArray())
        {
            script.Remove();
        }

After that i am trying to use //text() to get all the text nodes.

foreach (HtmlTextNode node in Document.DocumentNode.SelectNodes("//text()"))
        {
            TempString.AppendLine(node.InnerText);
        }

However not only i am not getting just text i am also getting numerous /r /n characters.

Please i require a little guidance in this regard.

Answer 1

If you consider that script and style nodes only have text nodes for children, you can use this XPath expression to get text nodes that are not in script or style tags, so that you don't need to remove the nodes beforehand:

//*[not(self::script or self::style)]/text()

You can further exclude text nodes that are only whitespace using XPath's normalize-space():

//*[not(self::script or self::style)]/text()[not(normalize-space(.)="")]

or the shorter

//*[not(self::script or self::style)]/text()[normalize-space()]

But you will still get text nodes that may have leading or trailing whitespace. This can be handled in your application as @aL3891 suggests.

Answer 2

If \r \n characters in the final string is the problem, you could just remove them after the fact:

TempString.ToString().Replace("\r", "").Replace("\n", "");