WebRequest returning more source than Browser > vi

Question

I am writing a Selenium utility in C# to help with automated testing. I may be missing something completely obvious here, but why is it that if I send an HttpWebRequest off to a server and retrieve the response stream, I end up with more source (i.e more mappable WebElements for Selenium) than I get when I right-click - view page source in my browser?

This means that I am mapping elements that the Selenium driver cant find come runtime.

Me so confuse :S

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(res.GetResponseStream());

var src = sr.ReadToEnd();

Answer 1

There are a number of possibilities. One likely culprit is the difference in the information you're sending to the server. Browsers typically send various headers, cookies, etc., that a web request does not have unless you explicitly add it.

For example, when I hit www.google.com with my web browser, I get stuff from Google Plus, and I get the fanciest experience possible because I'm on an evergreen browser. Roughly 139000 characters appear in my View Source page.

However, when I do a web request to the same URL (using the following code), I get only 45000 characters in the response stream:

async void Main()
{
    var result = await GetTextAsync("https://www.google.com");
    Console.Write(result.Length);
}

public async Task<string> GetTextAsync(string url){
    var result = await WebRequest.Create(url).GetResponseAsync();
    using (var stream = result.GetResponseStream())
    using (var reader = new StreamReader(stream))
    {
        return await reader.ReadToEndAsync();
    }
}

I suspect that if I were to set the WebRequest to send all the same cookies and headers that Chrome is sending to Google, my results would be much more similar.