i know that the topic has been discussed many times
but I need to understand how to write code in the correct way.
I use more times the same HttpWebRequest (to the same url) with protocol version HTTP 1.1.
Method = "POST"
KeepAlive = True
But everytime i need to send a different request, and get a different response.
(NB. This next code, it's not correct, and throw an exception)
Private Sub SendHttpWebReq()
Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
httpWebReq.Method = "POST"
httpWebReq.KeepAlive = True
httpWebReq.ContentType = "application/x-www-form-urlencoded"
Dim myRequestString As New List(Of String) From {"abc", "def"}
Dim ContentList As New List(Of String)
For a = 0 To 1
Dim inputData As String = MyRequestString(a)
Dim postData As String = "firstone" + ChrW(61) + inputData
Dim encoding As New System.Text.ASCIIEncoding()
Dim byteData As Byte() = encoding.GetBytes(postData)
httpWebReq.ContentLength = byteData.Length
Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
newStream.Write(byteData, 0, byteData.Length)
newStream.Flush()
newStream.Dispose()
Dim Response As Net.WebResponse = httpWebReq.GetResponse()
Dim ResponseStream As Io.Stream = Response.GetResponseStream()
Dim Content = New Io.MemoryStream()
ResponseStream.CopyTo(Content)
Response.Close()
Response.Dispose()
ResponseStream.Flush()
ResponseStream.Dispose()
ContentList.Add(System.Text.Encoding.UTF8.GetString(Content.ToArray))
Content = Nothing
Next
End Sub
When i run the code, the first time i get the correct response, but when i try to 'reuse' the HttpWebRequest, an Exception it's thrown
at the line:
httpWebReq.ContentLength = byteData.Length
the exception it's 'This property cannot be set after writing has started'
searching i've found this topic:
Am I able to reuse a HttpWebRequest?
Where it's explained that for 'reuse' an HttpWebRequest, the stream and WebResponse must be closed, and i made it, releasing the resources.
Also in this topic it's explained the same thing: https://social.msdn.microsoft.com/Forums/en-US/8efad6b3-bc75-48f0-9858-8115dabb85c8/reusing-httpwebrequest-object?forum=netfxnetcom
But in this other topic: This property cannot be set after writing has started! on a C# WebRequest Object
a member says that it's not possible to reuse the HttpWebRequest.
I'm in confusion between 'reuse' and 'create a new one'.
And i need to understand what 'KeepAlive' it's referred to: to the connection, or to the Request?
I suppose that when i execute this instruction:
Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
I should create an instance of HttpWebRequest class,
But i should establish the connection with this instruction
Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
Am i correct?
Any help?
This is what I think needs some clarifications, because this statement may be considered misleading, the way it's phrased:
1 WebRequest => 1 WebResponse. You can't change anything in a WebRequest once it has been initialized.
This remains valid, in principle, but the initialized term could be confusing. A better definition, would be:
You can't change any parameter of a WebRequest after the request has been issued and a WebResponse has been returned, until after the WebResponse is closed (disposed).
After a WebResponse has returned a result, it can be closed - explicitly or implicitly (in a Using block) - and you can request another, modifying as necessary the WebRequest parameters (e.g. changing the Method from POST to GET).
More, a WebRequest must be re-initialized before requesting a new WebResponse. If you don't, it just falls back to it's defaults.
The code I posted below, is an example of a classic context (a Web LogIn request) when a WebRequest must be re-initialized multiple times in the same procedure, to receive an undetermined number of WebResponses until a destination address (or landing page) is reached.
This is, more or less, the schema:
--------------
(GET or POST) | WebRequest | (Method is POST)
|---------> | GET/(POST) | <-----------| <-------------- |
| -------------- | |
| | | |
-------------- --------------- ------------------ --------------
| New | | WebResponse |--> | LogIn Required |-->| LogIn |
| Location | --------------- ------------------ | Address |
| (Referer | | --------------
| Set) | |
-------------- (Set Cookies)
| |
| ---------------
| | LogIn |
Redirection <----| OK |---NO---|
--------------- |
| |
YES |
(Set Cookies) |
| Request
--------------- Denied
| Response | |
| URI | |
--------------- |
| |
EXIT <------------|
|
Note that, issuing a WebRequest, if accessing the requested resource URI requires authentication, the server may NOT answer with a StatusCode 302 (Found), 301 (Moved) or 303 (Redirected), it might just set the StatusCode to 200 (OK). The redirection is implicit because the "Location" Header is set or, if it's a WebForm Login, the Html page retrieved contains the redirection.
Anyway, after a redirection has been detected, the new redirected location must be followed to destination. A redirection may consist of one or more Hops, which often must be addressed manually (to verify that we're sent where we actually want to go).
About the keep-alive Header.
A
keep-alive header is set by the Client and/or the Server, to
hint the counterpart that the established
connection should be maintained open, at least for some time, because the chance that other resources, linked to the current transaction, will be exchanged is high.
This prevents the creation of a possibly high number of costly connections.
This setup is common in
Http and
Ftp requests and it's the standard in
Http 1.1.
Hypertext Transfer Protocol (HTTP) Keep-Alive Header (IETF)
Compatibility with HTTP/1.0 Persistent Connections (IETF)
It needs to be remembered that the keep-alive header is referring to the Connection that has been established with a WebRequest, not to the WebRequest itself. When a connection is created specifing that it should be kept open, subsequent requests should maintain the connection: keep-alive Header to conform with the protocol.
In a .NET HttpWebRequest, setting the KeepAlive property to False, is equivalent to setting the connection: close Header.
However, the logic which governs a Connection and the Connection-Pool which a process is granted access to, is managed by the ServicePointManager, using a ServicePoint as a reference for every connection request.
Thus, a WebRequest may specify that the Connection it is asking to create should be kept open (because it needs to be reused more times), but the real logic behind a Connection, how it is established, maintained and managed, lies somewhere else.
In Http 2.0 (StackOverflow/StackExchange use this protocol), the keep-alive setting is completely ignored, for this exact reason. A Connection logic is managed at a higher, independent, level.
(...) when you call 1 new HttpWebRequest every 3 seconds, every 10
seconds, or every 60 seconds? What's the difference when i send those
requests with True or False?
You set the KeepAlive property to hint the Connection Manager that the Connection established should be kept open, because you know it will be re-used. However, both the ServicePointManager and the remote Server will comply to the request if the protocol in use provides for this setting and within the limits imposed by the internal logic that governs the complex of the Connection-Pools.
It should be set in Http 1.0, it's the default setting in Http 1.1, it's ignored in Http 2.0.
Since you can't know which protocol will be used until a Connection is established, it's usually set to keep-alive, because some devices (Proxies, specifically) in the route to the requested resource, might need this setting to be explicit (read the IETF documents about Proxies and their behaviour).
In this example, a
WebRequest is repeatedly initialized in a Loop, and the underlying
WebResponse is
Disposed each time, until a
StatusCode 200 (OK) is received or the request is denied or we have been redirected too many times (a Cancellation Token may also be useful here).
In the example, the main method is meant to be called this way:
Public Async Sub SomeMethodAsync()
LoginParameters = New LoginObject() With {
.CookieJar = New CookieContainer,
.LogInUrl = "[Some IP Address]",
.Credentials = New Dictionary(Of String, String)
}
LoginParameters.Credentials.Add("UserName", "[Username]")
LoginParameters.Credentials.Add("Email", "[email]")
LoginParameters.Credentials.Add("Password", "[Password]")
LoginParameters = Await HttpLogIn(LoginParameters)
End Sub
The LoginParameters object must be preserved, because references a CookieContainer, which contains the Cookies received after the authentication. These Cookies are passed to the server when a new WebRequest is initialized, as a "proof" that the request's credentials are already authenticated. Note that these Cookies expire after a while (thay are "refreshed" when a new WebRequest is issued, unless the Session has a time limit). If this is the case, the Login procedure is automatically repeated.
Imports System.Net
Imports System.Net.Security
Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Imports System.Security.Cryptography.X509Certificates
Imports System.Text
Public LoginParameters As LoginObject
Public Class LoginObject
Public Property LogInUrl As String
Public Property ResponseUrl As String
Public Property Credentials As Dictionary(Of String, String)
Public Property StatusCode As HttpStatusCode
Public Property CookieJar As New CookieContainer()
End Class
Public Async Function HttpLogIn(LogInParameters As LoginObject) As Task(Of LoginObject)
Dim httpRequest As HttpWebRequest
Dim StatusCode As HttpStatusCode
Dim MaxHops As Integer = 20
' Windows 7 (.Net 4.5.1+ required):
'ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
' Windows 10 (.Net 4.5.1+ required):
ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault
'If needed or for testing
'ServicePointManager.ServerCertificateValidationCallback = AddressOf CertificateValidation
httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
Try
HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
StatusCode = httpResponse.StatusCode
End Using
If StatusCode = HttpStatusCode.OK OrElse StatusCode = HttpStatusCode.NoContent Then
'POST Parameters are URLEncoded and the encoded strings converted to a Byte array of UTF8 chars
Dim EncodedParameters As Byte() = HTTP_EncodePOSTParameters(LogInParameters.Credentials)
httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
httpRequest.Method = WebRequestMethods.Http.Post
httpRequest.ContentType = "application/x-www-form-urlencoded"
httpRequest.ContentLength = EncodedParameters.Length
HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
Using stream As Stream = Await httpRequest.GetRequestStreamAsync()
stream.Write(EncodedParameters, 0, EncodedParameters.Length)
End Using
Dim Hops As Integer = 0
Dim Referer As String = LogInParameters.LogInUrl
Dim LastHttpMethod As String = httpRequest.Method
Do
'Evaluate Authentication redirect or page moved
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
StatusCode = httpResponse.StatusCode
LogInParameters.ResponseUrl = URIFromResponseLocation(httpResponse).ToString()
End Using
If (StatusCode = HttpStatusCode.Moved) OrElse
(StatusCode = HttpStatusCode.Found) OrElse
(StatusCode = HttpStatusCode.RedirectMethod) OrElse
(StatusCode = HttpStatusCode.RedirectKeepVerb) Then
httpRequest = WebRequest.CreateHttp(LogInParameters.ResponseUrl)
HTTP_RequestHeadersInit(httpRequest, Referer, LogInParameters.CookieJar)
If StatusCode = HttpStatusCode.RedirectKeepVerb Then
httpRequest.Method = LastHttpMethod
Else
LastHttpMethod = httpRequest.Method
End If
End If
If (CType(StatusCode, Integer) > 320) OrElse Hops >= MaxHops Then
Exit Do
End If
Hops += 1
Loop While (StatusCode <> HttpStatusCode.OK)
If StatusCode = HttpStatusCode.OK Then
LogInParameters.CookieJar = httpRequest.CookieContainer
End If
End If
Catch exW As WebException
StatusCode = If(exW.Response IsNot Nothing,
CType(exW.Response, HttpWebResponse).StatusCode,
CType(exW.Status, HttpStatusCode))
Catch exS As System.Exception
StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)
Finally
ServicePointManager.ServerCertificateValidationCallback = Nothing
End Try
LogInParameters.StatusCode = StatusCode
Return LogInParameters
End Function
Private Sub HTTP_RequestHeadersInit(ByRef httpReq As HttpWebRequest,
Referer As String,
CookiesJar As CookieContainer)
httpReq.Date = DateTime.Now
httpReq.CookieContainer = CookiesJar
httpReq.KeepAlive = True
httpReq.ConnectionGroupName = Guid.NewGuid().ToString()
httpReq.AllowAutoRedirect = False
httpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
httpReq.ServicePoint.Expect100Continue = False
httpReq.Referer = Referer
httpReq.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
httpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
httpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
httpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
httpReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
End Sub
Private Function HTTP_EncodePOSTParameters(PostParameters As Dictionary(Of String, String)) As Byte()
Dim Encoder As New System.Text.UTF8Encoding()
Dim CredentialValues As New StringBuilder()
Dim _first As Boolean = True
For Each CurrentKeyPair As KeyValuePair(Of String, String) In PostParameters
If _first = False Then CredentialValues.Append("&")
CredentialValues.AppendFormat("{0}={1}", WebUtility.UrlEncode(CurrentKeyPair.Key),
WebUtility.UrlEncode(CurrentKeyPair.Value))
_first = False
Next
Return Encoder.GetBytes(CredentialValues.ToString())
End Function
Private Function URIFromResponseLocation(Response As HttpWebResponse) As System.Uri
Dim uri As Uri
Dim Location As String = Response.Headers("Location")
Try
If uri.IsWellFormedUriString(Location, UriKind.Absolute) Then
uri = New Uri(Location, UriKind.Absolute)
Else
Dim HostUri As String = Response.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
UriFormat.Unescaped) + Location
uri = If(uri.IsWellFormedUriString(HostUri, UriKind.Absolute),
New Uri(HostUri),
New Uri(Response.ResponseUri.GetComponents(UriComponents.Scheme, UriFormat.Unescaped) +
Response.ResponseUri.Host + Location))
End If
Catch ExceptionOnInvalidUri As Exception
uri = New Uri(Location, UriKind.Relative)
End Try
Return uri
End Function
Private Function CertificateValidation(sender As Object,
CACert As X509Certificate,
CAChain As X509Chain,
PolicyErrors As SslPolicyErrors) As Boolean
'This method, as it is, accepts a Server certificate in any case
'It could be eventually adapted to refuse a connection (returning false)
'if the certificate is invalid, expired or from a untrusted path
If (PolicyErrors = SslPolicyErrors.None) Then Return True
'If a Certificated must be added to the Chain, uncomment the code below,
'selecting a Certificate in the Local (or other) Storage
'Dim MyCert As X509Certificate2 = New X509Certificate2("[localstorage]/[ca.cert]")
'CAChain.ChainPolicy.ExtraStore.Add(MyCert)
'CAChain.Build(MyCert)
'For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
' If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
' (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
' Return False
' End If
'Next
Return True
End Function
