ReadFileEx, variable length - a few questions

Question

I'm trying to read from stderr of a child process. The data is lines of text created with sprintf(stderr, "some debug info\n"). I'm using ReadFileEx with a completion routine. I don't know how many lines of text or how long each line will be. So, what do I put as the nNumberOfBytesToRead parameter?

My guess is I put the max size of my buffer, which I will make 4k; although I don't know if that's an optimal size. I'm guessing that if the line written to stderr is shorter than 4k, the completion routine won't fire. I'm guessing that when the 4k is reached but more data remains, I have to fire off another ReadFileEx within the completion routine. I'll know this is the case because GetLastError will return ERROR_MORE_DATA. I'm hoping that I get a call when the buffer isn't full, but the child process has exited. I'm not sure I get a completion callback when the child process exits, because I passed the stderr write handle to the child when I created it; maybe I get the callback when I close that handle. Are there any race conditions when the child closes wrt to my reading stderr?

Here is the psuedo code of how the process and handles are created:

Attr.bInheritHandle = true
CreatePipe(&hr, &hw, &Attr, 0) and SetHandleInformation(hX, HANDLE_FLAG_INHERIT) on hX the child uses.
Si.hStdXXX = handles from CreatePipe that child uses
CreateProcess(inherit=true, &Si)

Details (Tx extension is a wrapper that throws errors):

HANDLE Create() {
    STARTUPINFO SI, *pSI = NULL;
    bool fInherit = m_fInherit;
    if (m_fStdOut || m_fStdIn || m_fStdErr) {
        fInherit = true;
        SECURITY_ATTRIBUTES Attr;
        Attr.nLength = sizeof(SECURITY_ATTRIBUTES); 
        Attr.bInheritHandle = TRUE; 
        Attr.lpSecurityDescriptor = NULL;
        if (m_fStdOut) // Create a pipe for the child process's STDOUT. The child will use the write.
            CHandle::CreatePipe(m_hStdOutR, m_hStdOutW, &Attr, CP_INHERIT_WRITE);
        if (m_fStdErr) // Create a pipe for the child process's STDERR. The child will use the write.
            CHandle::CreatePipe(m_hStdErrR, m_hStdErrW, &Attr, CP_INHERIT_WRITE);
        if (m_fStdIn) // Create a pipe for the child process's STDIN. The child will use the read.
            CHandle::CreatePipe(m_hStdInR, m_hStdInW, &Attr, CP_INHERIT_READ);
        // Set up members of the STARTUPINFO structure. 
        // This structure specifies the STDIN and STDOUT handles for redirection.
        ZeroStruct(SI);
        SI.cb = sizeof(STARTUPINFO); 
        SI.hStdError = m_hStdErrW, SI.hStdOutput = m_hStdOutW, SI.hStdInput = m_hStdInR;
        SI.dwFlags |= STARTF_USESTDHANDLES;
        pSI = &SI;
    }
    //  m_fCpu, m_fNuma are masks to set affinity to cpus or numas
    CreateProcessTx(NULL, m_szCmdLine, fInherit, m_fFlags, pSI, &m_pi, m_fCpu, m_fNuma, 5);
    m_hProc = m_pi.hProcess;
    m_hThread = m_pi.hThread;
    if (!m_fThread)
        m_hThread.Close();
    return m_hProc;
}

static void CreatePipe(CHandle &hRead, CHandle &hWrite, SECURITY_ATTRIBUTES* pAttr, BYTE fInheritMask) {
    HANDLE hReadTmp = NULL, hWriteTmp = NULL;
    CreatePipeTx(hReadTmp, hWriteTmp, pAttr);
    SetHandleInformation(hReadTmp, HANDLE_FLAG_INHERIT, (fInheritMask&CP_INHERIT_READ) ? HANDLE_FLAG_INHERIT : 0); 
    SetHandleInformation(hWriteTmp, HANDLE_FLAG_INHERIT, (fInheritMask&CP_INHERIT_WRITE) ? HANDLE_FLAG_INHERIT : 0);
    hRead = hReadTmp;
    hWrite = hWriteTmp;
}

Answer 1

Anonymous pipes created with CreatePipe can't used asynchronously. From the Windows SDK documentation:

Asynchronous (overlapped) read and write operations are not supported by anonymous pipes. This means that you cannot use the ReadFileEx and WriteFileEx functions with anonymous pipes. >In addition, the lpOverlapped parameter of ReadFile and WriteFile is ignored when these >functions are used with anonymous pipes.

Basically CreatePipe doesn't accept a FILE_FLAG_OVERLAPPED flag, and asynchronous I/O requires that you use the flag when creating the file handle.

You'll have to use CreateNamedPipe to create named pipes. The question Overlapped I/O on anonymous pipe has answer with a link to a replacement function MyCreatePipeEx you can use.

Your completion port should receive a zero length read event after attempting to read from a pipe that has been closed on the other end.

To read a variable amount of data from the client process just issue read requests of whatever size you find convenient and be prepared to handle read events that are shorter than you requested. Don't interpret a short but non-zero length as EOF. Keep issuing read requests until you get a zero length read or an error.

Also WaitForMultipleObjects won't work with completion routines, as they're only called while the thread is in an alterable state. Use WaitForMultipleObjectEx with the bAlertable argument set to to true. This function will return WAIT_IO_COMPLETION after running one or more completion routines. In that case you'll probably want to immediately call WaitForMultipleObjectEx again.