Upper limit to UDP performance on windows server 2

It looks like from my testing I am hitting a performance wall on my 10gb network. I seem to be unable to read more than 180-200k packets per second. Looking at perfmon, or task manager I can receive up to a million packets / second if not more. Testing 1 socket or 10 or 100, doesn't seem to change this limit of 200-300k packets a second. I've fiddled with RSS and the like without success. Unicast vs multicast doesn't seem to matter, overlapped i/o vs synchronous doesn't make a difference either. Size of packet doesn't matter either. There just seems to be a hard limit to the number of packets windows can copy from the nic to the buffer. This is a dell r410. Any ideas?

#include "stdafx.h"

#include <WinSock2.h>
#include <ws2ipdef.h>

static inline void fillAddr(const char* const address, unsigned short port, sockaddr_in &addr)
{
    memset( &addr, 0, sizeof( addr ) );
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = inet_addr( address );
    addr.sin_port = htons(port);
}

int _tmain(int argc, _TCHAR* argv[])
{
#ifdef _WIN32
    WORD wVersionRequested;
    WSADATA wsaData;
    int err;

    wVersionRequested = MAKEWORD( 1, 1 );

    err = WSAStartup( wVersionRequested, &wsaData );
#endif
    int error = 0;
    const char* sInterfaceIP = "10.20.16.90";
    int nInterfacePort = 0;

    //Create socket
    SOCKET m_socketID = socket( AF_INET, SOCK_DGRAM, IPPROTO_UDP );

    //Re use address
    struct sockaddr_in addr;
    fillAddr( "10.20.16.90", 12400, addr ); //"233.43.202.1"

    char one = 1;
    //error = setsockopt(m_socketID, SOL_SOCKET, SO_REUSEADDR , &one, sizeof(one));
    if( error != 0 )
    {
        fprintf( stderr, "%s: ERROR setsockopt returned %d.\n", __FUNCTION__, WSAGetLastError() );
    }

    //Bind
    error = bind( m_socketID, reinterpret_cast<SOCKADDR*>( &addr ), sizeof( addr ) );

    if( error == -1 )
    {
        fprintf(stderr, "%s: ERROR %d binding to %s:%d\n",
            __FUNCTION__, WSAGetLastError(), sInterfaceIP, nInterfacePort);
    }

    //Join multicast group
    struct ip_mreq mreq;
    mreq.imr_multiaddr.s_addr = inet_addr("225.2.3.13");//( "233.43.202.1" );
    mreq.imr_interface.s_addr = inet_addr("10.20.16.90");

    //error = setsockopt( m_socketID, IPPROTO_IP, IP_ADD_MEMBERSHIP, reinterpret_cast<char*>( &mreq ), sizeof( mreq ) );

    if (error == -1)
    {
        fprintf(stderr, "%s: ERROR %d trying to join group %s.\n", __FUNCTION__, WSAGetLastError(), "233.43.202.1"  );
    }

    int bufSize = 0, len = sizeof(bufSize), nBufferSize = 10*1024*1024;//8192*1024;

    //Resize the buffer
    getsockopt(m_socketID, SOL_SOCKET, SO_RCVBUF, (char*)&bufSize, &len );
    fprintf(stderr, "getsockopt size before %d\n", bufSize );


    fprintf(stderr, "setting buffer size %d\n", nBufferSize );

    error =  setsockopt(m_socketID, SOL_SOCKET, SO_RCVBUF,
        reinterpret_cast<const char*>( &nBufferSize ), sizeof( nBufferSize ) );
    if( error != 0 )
    {
        fprintf(stderr, "%s: ERROR %d setting the receive buffer size to %d.\n",
            __FUNCTION__, WSAGetLastError(), nBufferSize );
    }

    bufSize = 1234, len = sizeof(bufSize);
    getsockopt(m_socketID, SOL_SOCKET, SO_RCVBUF, (char*)&bufSize, &len );
    fprintf(stderr, "getsockopt size after %d\n", bufSize );

    //Non-blocking
    u_long op = 1;
    ioctlsocket( m_socketID, FIONBIO, &op );

    //Create IOCP
    HANDLE iocp = CreateIoCompletionPort( INVALID_HANDLE_VALUE, NULL, NULL, 1 );
    HANDLE iocp2 = CreateIoCompletionPort( (HANDLE)m_socketID, iocp, 5, 1 );

    char buffer[2*1024]={0};

    int r = 0;

    OVERLAPPED overlapped; 
    memset(&overlapped, 0, sizeof(overlapped));

    DWORD bytes = 0, flags = 0;
//  WSABUF buffers[1];
//
//  buffers[0].buf = buffer;
//  buffers[0].len = sizeof(buffer);
//
//  while( (r = WSARecv( m_socketID, buffers, 1, &bytes, &flags, &overlapped, NULL )) != -121 )
    //sleep(100000);
    while( (r = ReadFile( (HANDLE)m_socketID, buffer, sizeof(buffer), NULL, &overlapped )) != -121 )
    {
        bytes = 0;
        ULONG_PTR key = 0;
        LPOVERLAPPED pOverlapped;

        if( GetQueuedCompletionStatus( iocp, &bytes, &key, &pOverlapped, INFINITE ) )
        {
            static unsigned __int64 total = 0, printed = 0;

            total += bytes;

            if( total - printed > (1024*1024) )
            {
                printf( "%I64dmb\r", printed/ (1024*1024) );
                printed = total;
            }
        }

    }

    while( r = recv(m_socketID,buffer,sizeof(buffer),0) )
    {
        static unsigned int total = 0, printed = 0;

        if( r > 0 )
        {
            total += r;

            if( total - printed > (1024*1024) )
            {
                printf( "%dmb\r", printed/ (1024*1024) );
                printed = total;
            }
        }
    }

    return 0;
}

I am using Iperf as the sender and comparing the amount of data received to the amount of data sent: iperf.exe -c 10.20.16.90 -u -P 10 -B 10.20.16.51 -b 1000000000 -p 12400 -l 1000

edit: doing iperf to iperf the performance is closer to 180k or so without dropping (8mb client side buffer). If I am doing tcp I can do about 200k packets/second. Here's what interesting though - I can do far more than 200k with multiple tcp connections, but multiple udp connections do not increase the total (I test udp performance with multiple iperfs, since a single iperf with multiple threads doesn't seem to work). All hardware acceleration is tuned on in the drivers.. It seems like udp performance is simply subpar?

I've been doing some UDP testing with similar hardware as I investigate the performance gains that can be had from using the Winsock Registered I/O network extensions, RIO, in Windows 8 Server. For this I've been running tests on Windows Server 2008 R2 and on Windows Server 8.

I've yet to get to the point where I've begun testing with our 10Gb cards (they've only just arrived) but the results of my earlier tests and the example programs used to run them can be found here on my blog.

One thing that I might suggest is that with a simple test like the one you show where there's very little work being done to each datagram you may find that old fashioned, synchronous I/O, is faster than the IOCP design. Whilst the IOCP design steps ahead as the workload per datagram rises and you can fully utilise the multiple threads.

Also, are your test machines wired back to back (i.e. without a switch) or do they run through a switch; if so, could the issue be down to the performance of your switch rather than your test machines? If you're using a switch, or have multiple nics in the server, can you run multiple clients against the server, could the issue be on the client rather than the server?

What CPU usage are you seeing on the sending and receiving machines? Have you looked at the machine's cpu usage with Process Explorer? This is more accurate than Task Manager. Which CPU is handling the nic interrupts, can you improve things by binding these to another cpu? or changing the affinity of your test program to run on another cpu? Is your IOCP example spreading its threads across multiple NUMA nodes or are you locking all of them to one node?

I'm hoping to get to run some more tests next week and will update my answer when I have done so.

Edit: For me the problem was due to the fact that the NIC drivers had "flow control" enabled and this caused the sender to run at the speed of the receiver. This had some undesirable "non-paged pool" usage characteristics and turning off flow control allows you to see how fast the sender can go (and the difference in network utilisation between the sender and receiver clearly shows how much data is being lost). See my blog posting here for more details.