AIO on OS X vs Linux - why it doesn't work on

Question

My question is really simple. Why the code below does work on Linux, and doesn't on Mac OS X 10.6.2 Snow Leopard.

To compile save the file to aio.cc, and compile with g++ aio.cc -o aio -lrt on Linux, and g++ aio.cc -o aio on Mac OS X. I'm using Mac OS X 10.6.2 for testing on a Mac, and Linux kernel 2.6 for testing on Linux.

The failure I see on OS X is aio_write fails with -1 and sets errno to EAGAIN, which simply means "Resource temporarily unavailable". Why is that?

extern "C" {
#include <aio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <errno.h>
#include <signal.h>
}
#include <cassert>
#include <string>
#include <iostream>

using namespace std;

static void
aio_completion_handler(int signo, siginfo_t *info, void *context)
{
  using namespace std;
  cout << "BLAH" << endl;
}


int main()
{
  int err;

  struct sockaddr_in sin;
  memset(&sin, 0, sizeof(sin));

  sin.sin_port = htons(1234);
  sin.sin_addr.s_addr = inet_addr("127.0.0.1");
  sin.sin_family = PF_INET;

  int sd = ::socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
  if (sd == -1) {
    assert(!"socket() failed");
  }

  const struct sockaddr *saddr = reinterpret_cast<const struct sockaddr *>(&sin);
  err = ::connect(sd, saddr, sizeof(struct sockaddr));
  if (err == -1) {
    perror(NULL);
    assert(!"connect() failed");
  }

  struct aiocb *aio = new aiocb();
  memset(aio, 0, sizeof(struct aiocb));

  char *buf = new char[3];
  buf[0] = 'a';
  buf[1] = 'b';
  buf[2] = 'c';
  aio->aio_fildes = sd;
  aio->aio_buf = buf;
  aio->aio_nbytes = 3;

  aio->aio_sigevent.sigev_notify = SIGEV_SIGNAL;
  aio->aio_sigevent.sigev_signo = SIGIO;
  aio->aio_sigevent.sigev_value.sival_ptr = &aio;

  struct sigaction sig_act;

  sigemptyset(&sig_act.sa_mask);
  sig_act.sa_flags = SA_SIGINFO;
  sig_act.sa_sigaction = aio_completion_handler;

  sigaction(SIGIO, &sig_act, NULL);

  errno = 0;
  int ret = aio_write(aio);
  if (ret == -1) {
    perror(NULL);
  }  
  assert(ret != -1);  
}

UPDATE (Feb 2010): OSX does not support AIO on sockets at all. Bummer!

Answer 1

I have code very similar to yours on 10.6.2 (but writing to a file) working without any problems - so it is possible to do what you're trying.

Just out of curiosity, what value are you using for the SIGIO constant ? I found that an invalid value here in OS X would casue aio_write to fail - so I always pass SIGUSR1.

Maybe check the return value of sigaction() to verify the signal details?

Answer 2

OSX Allows you to use sockets via the (CF)RunLoop. Or getting callbacks from the runloop. That is the most elegant way I have found to use async IO on mac. You can use your existing socket and do a CFSocketCreateWithNative. And register callbacks on your runloop.

Here is a small snippet of code that shows how it can be setup, incomplete since I have cut down on a source file...

// This will setup a readCallback 
void SocketClass::setupCFCallback() {   
CFSocketContext     context = { 0, this, NULL, NULL, NULL };

if (CFSocketRef macMulticastSocketRef = CFSocketCreateWithNative(NULL, socketHandle_, kCFSocketReadCallBack,readCallBack, &context)) {
    if (CFRunLoopSourceRef macRunLoopSrc = CFSocketCreateRunLoopSource(NULL, macMulticastSocketRef, 0)) {
        if (!CFRunLoopContainsSource(CFRunLoopGetCurrent(), macRunLoopSrc, kCFRunLoopDefaultMode)) {
            CFRunLoopAddSource(CFRunLoopGetCurrent(), macRunLoopSrc, kCFRunLoopDefaultMode);
            macRunLoopSrc_ = macRunLoopSrc;
        }
        else
            CFRelease(macRunLoopSrc);
    }
    else
        CFSocketInvalidate(macMulticastSocketRef);
    CFRelease(macMulticastSocketRef);
}
}



void SocketClass::readCallBack(CFSocketRef inref, CFSocketCallBackType  type,CFDataRef , const void *, void *info) {
if (SocketClass*    socket_ptr = reinterpret_cast<SocketClass*>(info))
    socket_ptr->receive(); // do stuff with your socket

}

Answer 3

The points raised in your links all point to a different method for raising io completion notifications (e.g. kqueue which is a BSD specific mechanism), but doesn't really answer your question re POSIX methods for async io. and whether they work on Darwin.

The UNIX world really is a mish mash of solutions for this, and it would be really good if there was one tried and tested solutiom that worked across all platforms, alas currently there's not - POSIX being the one that aims for the most consistency.

It's a bit of a stab in the dark, but it might be useful as well to set nonblocking on your socket handle ( i.e. set socket option O_NONBLOCK ) as well as using SIGUSR1

If I get some time I'll work with your socket sample and see if I can get anything out of that too.

Best of luck.

Answer 4

The presented code was tested on Mountain Lion 10.8.2. It works with a small correction. The line
"aio->aio_fildes = sd;"

should be changed for example to:
aio->aio_fildes = open( "/dev/null", O_RDWR);

to get the expected result.

see manual. "The aio_write() function allows the calling process to perform an asynchronous write to a previously opened file."