Get web page using HtmlAgilityPack.NETCore

Question

I used the HtmlAgilityPack for work with html pages. Previously I did this:

HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(url);
var nodes = document.DocumentNode.SelectNodes("necessary node");

but now i need to use the HtmlAgilityPack.NETCore where HtmlWeb is absent. What should i use instead HtmlWeb to have the same result?

Answer 1

You can use HttpClient to get the content of the page.

Answer 2

I had the same problem in Visual Studio code with netcoreapp1.0. Ended up using HtmlAgilityPack version 1.5.0-beta5 instead.

Remember to add:

using HtmlAgilityPack;
using System.Net.Http;
using System.IO;

I did it like this:

HttpClient hc = new HttpClient(); 
HttpResponseMessage result = await hc.GetAsync($"http://somewebsite.com"); 
Stream stream = await result.Content.ReadAsStreamAsync(); 
HtmlDocument doc = new HtmlDocument(); 
doc.Load(stream); 
HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@class='whateverclassyouarelookingfor']");

Answer 3

Use the HttpClient as a new way to interact with remote resources via http.

As for your solution, you probably need to use the async methods here for non-blocking your thread, instead of .Result usage. Also note that HttpClient was meant to be used from different threads starting from .Net 4.5, so you should not recreate it each time:

// instance or static variable
HttpClient client = new HttpClient();

// get answer in non-blocking way
using (var response = await client.GetAsync(url))
{
    using (var content = response.Content)
    {
        // read answer in non-blocking way
        var result = await content.ReadAsStringAsync();
        var document = new HtmlDocument();
        document.LoadHtml(result);
        var nodes = document.DocumentNode.SelectNodes("Your nodes");
        //Some work with page....
    }
}

Great article about async/await: Async/Await - Best Practices in Asynchronous Programming by @StephenCleary | March 2013

Answer 4

I wrote this and it's working. Is this a good way to solve my problem?

using (HttpClient client = new HttpClient())
{
    using (HttpResponseMessage response = client.GetAsync(url).Result)
    {
        using (HttpContent content = response.Content)
        {
            string result = content.ReadAsStringAsync().Result;
            HtmlDocument document = new HtmlDocument();
            document.LoadHtml(result);
            var nodes = document.DocumentNode.SelectNodes("Your nodes");
            //Some work with page....
        }
    }
}