Parsing HTML document: Regular expression or LINQ?

2019-06-16 20:25发布

站内文章 / C#
38 0 0

问题:

Trying to parse an HTML document and extract some elements (any links to text files).

The current strategy is to load an HTML document into a string. Then find all instances of links to text files. It could be any file type, but for this question, it's a text file.

The end goal is to have an IEnumerable list of string objects. That part is easy, but parsing the data is the question.

<html>
<head><title>Blah</title>
</head>
<body>
<br/>
<div>Here is your first text file: <a href="http://myServer.com/blah.txt"></div>
<span>Here is your second text file: <a href="http://myServer.com/blarg2.txt"></span>
<div>Here is your third text file: <a href="http://myServer.com/bat.txt"></div>
<div>Here is your fourth text file: <a href="http://myServer.com/somefile.txt"></div>
<div>Thanks for visiting!</div>
</body>
</html>

The initial approaches are:

  • load the string into an XML document, and attack it in a Linq-To-Xml fashion.
  • create a regex, to look for a string starting with href=, and ending with .txt

The question being:

  • what would that regex look like? I am a regex newbie, and this is part of my regex learning.
  • which method would you use to extract a list of tags?
  • which would be the most performant way?
  • which method would be the most readable/maintainable?

Update: Kudos to Matthew on the HTML Agility Pack suggestion. It worked just fine! The XPath suggestion works as well. I wish I could mark both answers as 'The Answer', but I obviously cannot. They are both valid solutions to the problem.

Here's a C# console app using the regex suggested by Jeff. It reads the string fine, and will not include any href that is not ended with .txt. With the given sample, it correctly does NOT include the .txt.snarg file in the results (as provided in the HTML string function).

using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
using System.IO;

namespace ParsePageLinks
{
    class Program
    {
        static void Main(string[] args)
        {
            GetAllLinksFromStringByRegex();
        }

        static List<string> GetAllLinksFromStringByRegex()
        {
            string myHtmlString = BuildHtmlString();
            string txtFileExp = "href=\"([^\\\"]*\\.txt)\"";

            List<string> foundTextFiles = new List<string>();

            MatchCollection textFileLinkMatches = Regex.Matches(myHtmlString, txtFileExp, RegexOptions.IgnoreCase);
            foreach (Match m in textFileLinkMatches)
            {
                foundTextFiles.Add( m.Groups[1].ToString()); // this is your captured group
            }

            return files;
        }

            static string BuildHtmlString()
            {
                return new StringReader(@"<html><head><title>Blah</title></head><body><br/>
<div>Here is your first text file: <a href=""http://myServer.com/blah.txt""></div>
<span>Here is your second text file: <a href=""http://myServer.com/blarg2.txt""></span>
<div>Here is your third text file: <a href=""http://myServer.com/bat.txt.snarg""></div>
<div>Here is your fourth text file: <a href=""http://myServer.com/somefile.txt""></div>
<div>Thanks for visiting!</div></body></html>").ReadToEnd();
            }       
        }
    }

回答1:

I would recommend regex. Why?

  • Flexible (case-insensitivity, easy to add new file extensions, elements to check, etc.)
  • Fast to write
  • Fast to run

Regex expressions will not be hard to read, as long as you can WRITE regexes.

using this as the regular expression:

href="([^"]*\.txt)"

Explanation:

  • It has parentheses around the filename, which will result in a "captured group" which you can access after each match has been found.
  • It has to escape the "." by using the regex escape character, a backslash.
  • It has to match any character EXCEPT double-quotes: [^"] until it finds
    the ".txt"

it translates into an escaped string like this:

string txtExp = "href=\"([^\\\"]*\\.txt)\"

Then you can iterate over your Matches:

Matches txtMatches = Regex.Matches(input, exp, RegexOptions.IgnoreCase);
foreach(Match m in txtMatches) {
  string filename = m.Groups[1]; // this is your captured group
}


回答2:

Neither. Load it into an (X/HT)MLDocument and use XPath, which is a standard method of manipulating XML and very powerful. The functions to look at are SelectNodes and SelectSingleNode.

Since you are apparently using HTML (not XHTML), you should use HTML Agility Pack. Most of the methods and properties match the related XML classes.

Sample implementation using XPath:

    HtmlDocument doc = new HtmlDocument();
    doc.Load(new StringReader(@"<html>
<head><title>Blah</title>
</head>
<body>
<br/>
<div>Here is your first text file: <a href=""http://myServer.com/blah.txt""></div>
<span>Here is your second text file: <a href=""http://myServer.com/blarg2.txt""></span>
<div>Here is your third text file: <a href=""http://myServer.com/bat.txt""></div>
<div>Here is your fourth text file: <a href=""http://myServer.com/somefile.txt""></div>
<div>Thanks for visiting!</div>
</body>
</html>"));
        HtmlNode root = doc.DocumentNode;
        // 3 = ".txt".Length - 1.  See http://stackoverflow.com/questions/402211/how-to-use-xpath-function-in-a-xpathexpression-instance-programatically
        HtmlNodeCollection links = root.SelectNodes("//a[@href['.txt' = substring(., string-length(.)- 3)]]");
    IList<string> fileStrings;
    if(links != null)
    {
        fileStrings = new List<string>(links.Count);
        foreach(HtmlNode link in links)
        fileStrings.Add(link.GetAttributeValue("href", null));
    }
    else
        fileStrings = new List<string>(0);


回答3:

Alternatively to Matthew Flaschen's suggestion, DOM (e.g. if you suffer from a X?L allergy outbreak)

It gets a bad rep sometimes - I guess because implementations are funny sometimes, and the native COM interfaces are a bit unwieldy without some (minor) smart helpers, but I've found it a robust, stable and intuitive / explorable way to parse and manipulate HTML.



回答4:

REGEX is not fast, in fact it's slower than native string parse stuff in .NET. Don't believe me, see for yourself.

None of the examples above are faster than going to the DOM directly. 

HTMLDocument doc = wb.Document;
var links = doc.Links;


标签： c# regex linq parsing linq-to-xml
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~