Using NServiceBus in an Amazon EC2 hosted applicat

Question

We're starting a new project that has some messaging and queueing requirements that are pretty basic, but in the future there may be some additional requirements for things such as sagas to process messages that might arrive out of order and need to be resequenced.

We've just recently released a project that's completely hosted on Amazon EC2 which also has a simple messaging system in it. We have a very simplistic little pub/sub mechanism whereby we receive a message and then figure out which handler to use for the message based on its type. In our new project we also have a similar mechanism and may have some slightly more sophisticated requirements, but even if we could do away with having our own code to resolve the message handlers this would be cool too.

We're really keen to use NServiceBus as the pub/sub model is really nice, but up until now we've been using Amazon SQS as the queue provider. Each of our EC2 machines has a worker that's listening to the same SQS queue and pulling messages off of that to process them. Obviously SQS isn't supported as a transport layer (and I know it's because NServiceBus is built around a reliable and low latency queue) in NServiceBus.

I know that NServiceBus has a distributor, and I could imagine hosting that on its own EC2 instance, but the problem with that is that there is a huge single point of failure there, so we're basically screwed if that goes down. From what I gather, people setup windows failover clusters to deal with this problem for internal networks, but I don't know if this is applicable to EC2.

This was one of the only blog posts around that I could find by someone who had actually attempted to use NServiceBus in the cloud based context, and he doesn't seem to recommend it. I'm just wondering if anybody else on here has attempted it and if so would they have any advice to offer? Seems like such a shame that we won't be able to make use of such a great framework just because we're hosted on the cloud.

It looks like there's some progress being made with Azure Queues which is something we might look at, but for now we'd like to keep the infrastructure with Amazon as we have a lot of build automation that we can re-use that's based around that.

Answer 1

From experience of queueing in Amazon EC2, we've found SQS to be very sluggish for high performance messaging.

We're not currently using NServiceBus as the queueing interface but just thought I'd mention that we're successfully using RabbitMQ on Ubuntu instances in EC2 as our queue provider.

According to this post Udi Dahan suggests that people have swapped out MSMQ for RabbitMQ with NServiceBus so it might be a worthwhile investigation to use RabbitMQ on EC2.

There is a great guide on RabbitMQ's website describing an easy way of installing RabbitMQ on Amazon EC2 by launching a compatible Ubuntu image.

I'd highly recommend installing the RabbitMQ Management Plugin too which gives you a simple web admin interface on the queues.