IO Completion ports: separate thread pool to proce

Question

NOTE: I have added the C++ tag to this because a) the code is C++ and b) people using C++ may well have used IO completion ports. So please don't shout.

---

I am playing with IO completion ports, and have eventually fully understood (and tested, to prove) - both with help from RbMm - the meaning of the NumberOfConcurrentThreads parameter within CreateIoCompletionPort().

I have the following small program which creates 10 threads all waiting on the completion port. I tell my completion port to only allow 4 threads to be runnable at once (I have four CPUs). I then enqueue 8 packets to the port. My thread function outputs a message if it dequeues a packet with an ID > 4; in order for this message to be output, I have to stop at least one of the four currently running threads, which happens when I enter '1' at the console.

Now this is all fairly simple code. I have one big concern however, and that is that if all of the threads that are processing a completion packet get bogged down, it will mean no more packets can be dequeued and processed. That is what I am simulating with my infinite loop - the fact that no more packets are dequeued until I enter '1' at the console highlights this potential problem!

Would a better solution not be to have my four threads dequeuing packets (or as many threads as CPUs), then when one is dequeued, farm the processing of that packet off to a worker thread from a separate pool, thereby removing the risk of all threads in the IOCP being bogged down thus no more packets being dequeued?

I ask this as all the examples of IO completion port code I have seen use a method similar to what I show below, not using a separate thread pool which I propose. This is what makes me think that I am missing something because I am outnumbered!

Note: this is a somewhat contrived example, because Windows will allow an additional packet to be dequeued if one of the runnable threads enters a wait state; I show this in my code with a commented out cout call:

The system also allows a thread waiting in GetQueuedCompletionStatus to process a completion packet if another running thread associated with the same I/O completion port enters a wait state for other reasons, for example the SuspendThread function. When the thread in the wait state begins running again, there may be a brief period when the number of active threads exceeds the concurrency value. However, the system quickly reduces this number by not allowing any new active threads until the number of active threads falls below the concurrency value.

But I won't be calling SuspendThread in my thread functions, and I don't know which functions other than cout will cause the thread to enter a wait state, thus I can't predict if one or more of my threads will ever get bogged down! Hence my idea of a thread pool; at least context switching would mean that other packets get a chance to be dequeued!

#define _CRT_SECURE_NO_WARNINGS
#include <windows.h>
#include <thread>
#include <vector>
#include <algorithm>
#include <atomic>
#include <ctime>
#include <iostream>

using namespace std;

int main()
{
    HANDLE hCompletionPort1;
    if ((hCompletionPort1 = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 4)) == NULL)
    {
        return -1;
    }
    vector<thread> vecAllThreads;
    atomic_bool bStop(false);

    // Fill our vector with 10 threads, each of which waits on our IOCP.
    generate_n(back_inserter(vecAllThreads), 10, [hCompletionPort1, &bStop] {
        thread t([hCompletionPort1, &bStop]()
        {
            // Thread body
            while (true)
            {
                DWORD dwBytes = 0;
                LPOVERLAPPED pOverlapped = 0;
                ULONG_PTR uKey;
                if (::GetQueuedCompletionStatus(hCompletionPort1, &dwBytes, &uKey, &pOverlapped, INFINITE) == 1)
                {
                    if (dwBytes == 0 && uKey == 0 && pOverlapped == 0)
                        break;  // Special completion packet; end processing.

                    //cout << uKey; // EVEN THIS WILL CAUSE A "wait" which causes MORE THAN 4 THREADS TO ENTER!

                    if (uKey >4) 
                        cout << "Started processing packet ID > 4!" << endl;
                    while (!bStop)
                        ;   // INFINITE LOOP
                }
            }
        });
        return move(t);
    }
    );

    // Queue 8 completion packets to our IOCP...only four will be processed until we set our bool
    for (int i = 1; i <= 8; ++i)
    {
        PostQueuedCompletionStatus(hCompletionPort1, 0, i, new OVERLAPPED);
    }

    while (!bStop)
    {
        int nVal;
        cout << "Enter 1 to cause current processing threads to end: ";
        cin >> nVal;
        bStop = (nVal == 1);
    }
    for (int i = 0; i < 10; ++i)    // Tell all 10 threads to stop processing on the IOCP
    {
        PostQueuedCompletionStatus(hCompletionPort1, 0, 0, 0);  // Special packet marking end of IOCP usage
    }
    for_each(begin(vecAllThreads), end(vecAllThreads), mem_fn(&thread::join));

    return 0;
}

---

EDIT #1

What I mean by "separate thread pool" is something like the following:

class myThread {
public:
    void SetTask(LPOVERLAPPED pO) { /* start processing pO*/ }
private:
    thread m_thread;    // Actual thread object
};

// The threads in this thread pool are not associated with the IOCP in any way whatsoever; they exist
// purely to be handed a completion packet which they then process!
class ThreadPool
{
public:
    void Initialise() { /* create 100 worker threads and add them to some internal storage*/}
    myThread& GetNextFreeThread() { /* return one of the 100 worker thread we created*/}
} g_threadPool;

The code that each of my four threads associated with the IOCP then change to

if (::GetQueuedCompletionStatus(hCompletionPort1, &dwBytes, &uKey, &pOverlapped, INFINITE) == 1)
{
    if (dwBytes == 0 && uKey == 0 && pOverlapped == 0)
        break;  // Special completion packet; end processing.

    // Pick a new thread from a pool of pre-created threads and assign it the packet to process
    myThread& thr = g_threadPool.GetNextFreeThread();
    thr.SetTask(pOverlapped);

    // Now, this thread can immediately return to the IOCP; it doesn't matter if the
    // packet we dequeued would take forever to process; that is happening in the 
    // separate thread thr *that will not intefere with packets being dequeued from IOCP!*
}

This way, there is no possible way that I can end up in the situation where no more packets are being dequeued!

Answer 1

It seems there is conflicting opinion on whether a separate thread pool should be used. Clearly, as the sample code I have posted shows, there is potential for packets to stop being dequeued from the IOCP if the processing of the packets does not enter a wait state; given, the infinite loop is perhaps unrealistic but it does demonstrate the point.