How do I Create an HTTP Request Manually in .Net?

Question

I'd like to create my own custom HTTP requests. The WebClient class is very cool, but it creates the HTTP requests automatically. I'm thinking I need to create a network connection to the web server and pass my data over that stream, but I'm unfamiliar with the library classes that would support that kind of thing.

(Context, I'm working on some code for a web programming class that I'm teaching. I want my students to understand the basics of what's happening inside the "black box" of HTTP.)

Answer 1

Check out System.Net.Sockets.TcpClient if you want to write your own low level client. For HTTP GET's and POST's you can use the HttpWebRequest and HttpWebResponse classes though.

If you are really masochistic you could go lower than TcpClient and implement your own Socket, see the Socket class.

Answer 2

To really understand the internals of the HTTP protocol you could use TcpClient class:

using (var client = new TcpClient("www.google.com", 80))
{
    using (var stream = client.GetStream())
    using (var writer = new StreamWriter(stream))
    using (var reader = new StreamReader(stream))
    {
        writer.AutoFlush = true;
        // Send request headers
        writer.WriteLine("GET / HTTP/1.1");
        writer.WriteLine("Host: www.google.com:80");
        writer.WriteLine("Connection: close");
        writer.WriteLine();
        writer.WriteLine();

        // Read the response from server
        Console.WriteLine(reader.ReadToEnd());
    }
}

Another possibility is to activate tracing by putting the following into your app.config and just use WebClient to perform an HTTP request:

<configuration>
  <system.diagnostics>
    <sources>
      <source name="System.Net" tracemode="protocolonly">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="System.Net" value="Verbose"/>
    </switches>
    <sharedListeners>
      <add name="System.Net"
           type="System.Diagnostics.TextWriterTraceListener"
           initializeData="network.log" />
    </sharedListeners>
    <trace autoflush="true"/>
  </system.diagnostics>
</configuration>

Then you can perform an HTTP call:

using (var client = new WebClient())
{
    var result = client.DownloadString("http://www.google.com");
}

And finally analyze the network traffic in the generated network.log file. WebClient will also follow HTTP redirects.

Answer 3

Use the WebRequest or WebResponse classes, as required.

If you need to go lower level than these provide, look at the other System.Net.Sockets.*Client classes such as TcpClient.

Answer 4

Use WFetch for the demonstration.

As for programming, HttpWebRequest lets you control quite a bit about the request - again if it's for a demonstration I'd use Wireshark to sniff what's going over the wire when you do various tasks with the HttpWebRequest

Answer 5

In my response to this SO post SharePoint 2007, how to check if a folder exists in a document library I have included the basics of creating HttpWebRequest and reading response.

Edit: added a code example from the post above

System.Net.HttpWebRequest oReq; 
string sUrl = "http://yoursite/sites/somesite/DocumentLibrary"; 
oReq = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(sUrl); 

oReq.Method = "GET"; 
oReq.Credentials = System.Net.CredentialCache.DefaultCredentials; 
oReq.AllowAutoRedirect = true; 

//now send the request
using (System.IO.StreamWriter oRequest = 
        new System.IO.StreamWriter(oReq.GetRequestStream())) { 
   oRequest.WriteLine(); 
}

//and read all the response
using (System.Net.HttpWebResponse oResponse = oReq.GetResponse()){
  using (System.IO.StreamReader oResponseReader = 
        new System.IO.StreamReader(oResponse.GetResponseStream())){
    string sResponse = oResponseReader.ReadToEnd(); 
  }
}