Can't download webpage via C# webclient and vi

问题:

I want to download webpages html code, but have problems with several links. For example: http://www.business-top.info/, http://azerizv.az/ I recieve no html at all using this: 1. WebClient:

using (var client = new WebClient())
            {
                client.Encoding = System.Text.Encoding.UTF8;
                string result = client.DownloadString(resultUrl);
                Console.WriteLine(result);
                Console.ReadLine();
            }

2. Http request/response

var request = (HttpWebRequest)WebRequest.Create(resultUrl);
            request.Method = "POST";
            using (var response = (HttpWebResponse)request.GetResponse())
            {
                using (var stream = response.GetResponseStream())
                {
                    StreamReader sr = new StreamReader(stream, Encoding.UTF8);
                    string data = sr.ReadToEnd();
                    Console.WriteLine(data);
                    Console.ReadLine();
                }
            }

There are many such links, so I can't download html manually just via sourse code of web page via browser

回答1:

Some pages load in stages. First they load the core of the page and only then they evaluate any JavaScript inside which loads further content via AJAX. To scrape these pages you will need more advanced content scraping libraries, than just simple HTTP request sender.

EDIT: Here is a question in SO about the same problem that you are having now: Jquery Ajax Web page scraping using c#