My problem is a bit complex and i don't know how to explain it best
I have a 125k public proxy list. As you can also guess, most of them are invalid
I would like to quickly test all of them
So i have written an application which spawns 250 concurrent fetching tasks by Parallel.ForEach
So i am constantly fetching the same page by 250 different proxies to see whether they work or not
Each task uses one of the proxies and fetches the same page and look at the source code
Whether source code is valid, it returns true or false
I have set maximum allowed concurrent connection count per host to 1k
ServicePointManager.DefaultConnectionLimit = 1000;
//This sets to maximum number of concurrent connections to same host
So after few minutes fetching started, i am not able to connect any pages through my web browser. It shows in the below bar as resolving host.
I am not sure how to debug what is the exact problem?
I am suspicious that somehow DNS resolving is get broken. Or some other error happens. Any ideas are welcomed
My working environment is : c# .NET 4.6.2, Windows 8.1, 25 MBs fiber connection
I am using the below fetching function
public static cs_HttpFetchResults func_fetch_Page(string srUrl, int irTimeOut = 60,
string srRequestUserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0",
string srProxy = null,
int irCustomEncoding = 0,
bool blAutoDecode = true, bool blKeepAlive = true) {
cs_HttpFetchResults mycs_HttpFetchResults = new cs_HttpFetchResults();
mycs_HttpFetchResults.srFetchingFinalURL = srUrl;
try {
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(srUrl);
request.CookieContainer = new System.Net.CookieContainer();
if (srProxy != null) {
string srProxyHost = srProxy.Split(':')[0];
int irProxyPort = Int32.Parse(srProxy.Split(':')[1]);
System.Net.WebProxy my_awesomeproxy = new WebProxy(srProxyHost, irProxyPort);
my_awesomeproxy.Credentials = new NetworkCredential();
request.Proxy = my_awesomeproxy;
}
else {
request.Proxy = null;
}
request.Timeout = irTimeOut * 1000;
request.UserAgent = srRequestUserAgent;
request.KeepAlive = blKeepAlive;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("Accept-Language", "en-gb,en;q=0.5");
myWebHeaderCollection.Add("Accept-Encoding", "gzip, deflate");
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
using(WebResponse response = request.GetResponse()) {
using(Stream strumien = response.GetResponseStream()) {
Encoding myEncoding = Encoding.UTF8;
using(StreamReader sr = new StreamReader(strumien, myEncoding)) {
mycs_HttpFetchResults.srFetchBody = sr.ReadToEnd();
mycs_HttpFetchResults.blResultSuccess = true;
}
}
}
}
catch {
}
return mycs_HttpFetchResults;
}
After a while i have noticed that service host local system becomes a lot bigger in terms of ram usage