PLINQ: how to run a ParallelQuery on more than 4 t

Question

Update - changed the title of the question to reflect what I'm really after

Consider the following piece of code:

// this query generates 12 instances of Func<int>, which each when executed
// print something to the console and wait for 1 second.
var actions = Enumerable.Range(0, 12).Select(i => new Func<int>(() =>
{
    Console.WriteLine("{0} - waiting 1 sec", i);
    Thread.Sleep(1000);
    return 1;
}));

// define a parallel query. Note the WithDegreeOfParallelism call here.
var query = from action in actions.AsParallel().WithDegreeOfParallelism(12)
            select action();

// execute, measuring total duration
var stopw = Stopwatch.StartNew();
query.ToList();
Console.WriteLine(stopw.Elapsed);
Console.WriteLine(Environment.ProcessorCount); // 3 on my machine

When omitting the call to WithDegreeOfParallelism, this executes in 4 chunks, taking about 4 seconds in total, which is what I would expect since my CPU count is 3.

However, when calling WithDegreeOfParallelism with any number above 4, I always get 3 chunks, and the total duration does not go under 3 seconds. I would expect that a value of 12 would get a total duration of (a little more than) 1 second.

What am I missing? And how can I enforce the parallel execution of more than 4 non-CPU intensive tasks, which is what I'm after?

Update: I could of course go back to manually spinning up threads, but I was hoping that the new PFX library would make this a bit easier... Anyway, the code below gives me about 1 second total execution time

List<Thread> threads = new List<Thread>();
for (int i = 0; i < 12; i++)
{
    int i1 = i;
    threads.Add(new Thread(() =>
    {
        Console.WriteLine(i1);
        Thread.Sleep(1000);
    }));
}
threads.ForEach(t => t.Start());
threads.ForEach(t => t.Join());

Answer 1

Try starting new tasks in your parallel loop with the option TaskCreationOptions.LongRunning. They will start right away, instead of waiting until a thread on the threadpool becomes available.

Answer 2

WithDegreeOfParallelism dictates how many Tasks should PLINQ create, but not necessarily how many threads will be used.

Since Tasks execute as work items on the ThreadPool, the number of threads executing the query will be limited the size of the ThreadPool. The ThreadPool will add threads as necessary, but it may take a while - ThreadPool may add 2 threads per second or so.

If you want to add threads to the ThreadPool quickly, you can use the SetMinThreads method. If you put this code at the beginning of your code, the test should complete in a second or so:

int prevThreads, prevPorts;
ThreadPool.GetMinThreads(out prevThreads, out prevPorts);
ThreadPool.SetMinThreads(12, prevPorts);

You can decide how many threads you need, and then use SetMinThreads and SetMaxThreads to set bounds on the ThreadPool size.

Answer 3

As I said WithDegreeOfParallelism is setting only a upper bound.Try increasing your tasks from 10 to 100. You will ended up with around 10 seonds for all 100 of them. Your code is good for larger number of tasks having smaller operations. and add Console.WriteLine("{0} threads " ,Process.GetCurrentProcess().Threads.Count); inside your task then you can see how many threads are created.( Thread count is not the count of plinq created threads. See how it increasing).

There are lots of ways to do the parallelism with PLinq . Read this article http://msdn.microsoft.com/en-us/library/dd997411.aspx. You need to choose best way for the relevant requirement to get better performance.