For a couple of days I am working on a WebBrowser based webscraper. After a couple of prototypes working with Threads and DocumentCompleted events, I decided to try and see if I could make a simple, easy to understand Webscraper.
The goal is to create a Webscraper that doesn't involve actual Thread objects. I want it to work in sequential steps (i.e. go to url, perform action, go to other url etc. etc.).
This is what I got so far:
public static class Webscraper
{
private static WebBrowser _wb;
public static string URL;
//WebBrowser objects have to run in Single Thread Appartment for some reason.
[STAThread]
public static void Init_Browser()
{
_wb = new WebBrowser();
}
public static void Navigate_And_Wait(string url)
{
//Navigate to a specific url.
_wb.Navigate(url);
//Wait till the url is loaded.
while (_wb.IsBusy) ;
//Loop until current url == target url. (In case a website loads urls in steps)
while (!_wb.Url.ToString().Contains(url))
{
//Wait till next url is loaded
while (_wb.IsBusy) ;
}
//Place URL
URL = _wb.Url.ToString();
}
}
I am a novice programmer, but I think this is pretty straightforward code. That's why I detest the fact that for some reason the program throws an NullReferenceException at this piece of code:
_wb.Url.ToString().Contains(url)
I just called the _wb.Navigate() method so the NullReference can't be in the _wb object itself. So the only thing that I can imagine is that the _wb.Url object is null. But the while _wb.IsBusy() loop should prevent that.
So what is going on and how can I fix it?