Normally parallel processing is relevant only to CPU intensive operations. However, PLINQ specifically provides IO-intensive support using the WithDegreeOfParallelism extension. For example:
from site in new[]
{
"www.albahari.com",
"www.linqpad.net",
"www.oreilly.com",
"www.takeonit.com",
"stackoverflow.com",
"www.rebeccarey.com"
}
.AsParallel().WithDegreeOfParallelism(6)
let p = new Ping().Send (site)
select new
{
site,
Result = p.Status,
Time = p.RoundtripTime
}
But if supporting IO is the goal of WithDegreeOfParallelism, how then can PLINQ be further extended or used to achieve a "retry" effect, which is typical of IO operations? And what about a "delay" effect?
For example, if IO through a WCF service call throws a CommunicationException, I might want the same request made again with a "3 tries" strategy before moving on to the next resource. I might also want a minute wait between each try. And while I "wait" for a minute between each try, I don't want a thread blocked waiting.
In this MSDN article the author starts with a query similar to what I've shown above. He then transforms the query so that no threads are blocking. Unfortunately he lost the WithDegreeOfParallelism in the process. And either way, he did not address the issue of "retries" when an error occurs.
Anyone have or know of a slick way of doing this?
I was thinking of making a special IEnumerable wrapper that permitted values to be "re-inserted" while the collection was being walked by PLINQ. This would indeed cause a "retry" behavior, but it would still not allow for the "1 minute delay between retries" requirement. I could create the 1 minute delay with Thread.Sleep(), but I'm trying not to block threads.
Thoughts?
The article you linked actually shows how to avoid blocking threads for IO-bound operations by using a
Task
-based API (DownloadDataTask
) instead:Using PLINK /
Task.Run
/Task.Factory.StartNew
for IO-based operations is an anti-pattern. PLINQ (same asParallel.For
etc) is good for CPU-bound computational work, but there is no point in allocating and blocking a thread for a naturally asynchronous network/IO-bound operation, which doesn't need a thread at all while "in-flight". To follow the sample code you showed, that would be something likenew Ping().SendAsync(site)
, returning aTask
. You could then doawait Task.WhenAll(tasks)
and process the errors.Refer to "There Is No Thread" by Stephen Cleary, and his recent answer addressing the max degree of parallel IO. On top of that, it's quite easy to incorporate a retry logic, without getting any threads involved (for example, like this).