Choosing between .NET Service Bus Queues vs Azure

Question

Just a quick question regarding an Azure application. If I have a number of Web and Worker roles that need to communicate, documentation says to use the Azure Queue Service.

However, I've just read that the new .NET Service Bus now also offers queues. These look to be more powerful as they appear to offer a much more detailed API. Whilst the .NSB looks more interesting it has a couple of issues that make me wary of using it in distributed application. (for example, Queue Expiration... if I cannot guarantee that a queue will be renewed on time I may lose it all!).

Has anyone had any experience using either of these two technologies and could give any advice on when to choose one over the other.

I suspect that whilst the service bus looks more powerful, as my use case is really just enabling Web/Worker roles to communicate between each other, that the Azure Queue Service is what I'm after. But I'm just really looking for confirmation of that before progamming myself in to a corner :-)

Thanks in advance.

UPDATE

Have read up about the two systems over the break. It defo looks like .NET service bus is more specifically designed for integrating systems rather than providing a general purpose reliable messaging system. Azure Queues are distributed and so reliable and scalable in a way that .NSB queues are not and so more suitable for code hosted within Azure itself.

Thanks for the responses.

Answer 1

Azure Queues seams to match because your use case remains basic with simple REST-based Get/Put/Peek interface.

The documentation havs been updated recently (05/21/2015) and it details precisely when using the one or the other, and the common features (transaction support, size of the queue and message, Time to Live, ...) :

https://azure.microsoft.com/en-us/documentation/articles/service-bus-azure-and-service-bus-queues-compared-contrasted/

Answer 2

The queue related patterns a developer learns can be applied to both. Both can be used from a reliability and implementation-easiness point of view.

Things only Storage queue can do 1) The worker processing a message crashes. A subsequent worker wants to read the status of the message to continue from where the prior worker left off. 2) You require server side logs of all of the transactions executed against your queues.

But comparisons don't matter. If custom queue development is what we need then always use storage queue. It was the first to be developed by Microsoft. Service bus was brought-in copying BizTalk and the purpose is integration(hybrid), hence there are advanced features in this line: sessions, transactions, automatic dead-letters, etc.

This link provides a comparison , so does this link. It will be difficult to analyze everything and start in an agile development hence the above mentioned rule.

Answer 3

I would recommend that you stick with Azure Queues for communicating between web and worker roles. Using queues is the official and sanctioned way of communicating between Azure processes and I sincerely doubt that you will program yourself into a corner. Service Bus (AppFabric) has a higher overhead and although really good for talking to external apps, may not be optimal for quick and simple messages within your Azure app.

Answer 4

To make things very clear, this is a comparison between two Azure components, created at a different point in time, for different reasons.

Storage queues and Service Bus queues - compared and contrasted

Azure supports two types of queue mechanisms: Storage queues and Service Bus queues.

Storage queues, which are part of the Azure storage infrastructure, feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services.

Service Bus queues are part of a broader Azure messaging infrastructure that supports queuing as well as publish/subscribe, and more advanced integration patterns. For more information about Service Bus queues/topics/subscriptions, see the overview of Service Bus.

While both queuing technologies exist concurrently, Storage queues were introduced first, as a dedicated queue storage mechanism built on top of Azure Storage services. Service Bus queues are built on top of the broader messaging infrastructure designed to integrate applications or application components that may span multiple communication protocols, data contracts, trust domains, and/or network environments.

Answer 5

Storage Queues vs Service Bus

Here is a breakdown of some of the different considerations I had in thinking through this question.

Availability

Since the storage outage last November Azure promised they would never deploy code to all the regions at once again - built it into the system to make it impossible. https://azure.microsoft.com/en-us/blog/final-root-cause-analysis-and-improvement-areas-nov-18-azure-storage-service-interruption/

Here's what msdn says about availability:

If you are already using Azure Storage Blobs or Tables and you start using queues, you are guaranteed 99.9% availability. If you use Blobs or Tables with Service Bus queues, you will have lower availability.

Azure Queues are designed to support the decoupling of application components to increase scalability and tolerance for failures.

Development

Personally, I am comfortable with the storage APIs and already have need for its blob storage in other areas of most apps. Storage queues use the very same sdk as storage blobs. Azure Queues provide a uniform and consistent programming model across queues, tables, and BLOBs

Cost

The Receive and Delete mode supported by Service Bus provides the ability to reduce the messaging operation count (and associated cost) in exchange for lowered delivery assurance.

It seems like there is some tooling around cost control for service bus that you could leverage if you had to start keeping a budget to run your app - I did an attempt at breaking down the potential storage queue costs below :). It comes out to less than a hundred bucks a month at more than 40,000 queues an hour for GRS. Grouped in with the rest of our storage costs I don't see benefit to focusing on cutting costs here. (Bandwidth is the same for both and cancels itself out when comparing)

Storage pricing

You get unlimited free queues and operations - you pay for space

• assume 30K message size as an average
• assume 1000K in an MB not 1024
• assume you will not hit the graduated pricing above 1TB

30K / 1 message * 1 TB / 1000000000K * $.095 / 1 GB * 1000GB / 1 TB = $0.00000285 / message for the first TB of use

1 message / ~30K * 1000000000K / 1 TB = 33333333 messages in a TB

33333333 messages * $0.00000285 / message = ~$95 dollars for the first TB

spread out over a month we can do like 40,000 messages an hour with that 1st TB

Service Bus pricing

• 10 bucks a month base price
• pay per operation (any api call is an op) - add a queue / receive a queue / monitor the queue / etc.
• you get 12.5 million free ops / month
• pay per million ops after that

Hard to estimate usage here but 100 million operations costs 80 bucks a month

Batch Receive

Storage can batch up to a maximum of 32 messages by specifying message count when retrieving messages while Service Bus allows a queue client to batch multiple messages into a single send operation.

So storage is batch receive while service bus is batch send.

Monitoring

Azure Queues enable you to obtain a detailed log of all of the transactions executed against the queue, as well as aggregated metrics. This kind of support does not come out of box with Service Bus - but could probably find a prebuilt solution somewhere.

Forwarding

Service Bus has an auto-forwarding feature that storage queues is missing.

auto-forwarding enables thousands of queues to auto-forward their messages to a single queue, from which the receiving application consumes the message. You can use this mechanism to achieve security, control flow, and isolate storage between each message publisher.

Duplicates

The duplication detection functionality supported by Service Bus queues automatically removes duplicate messages sent to a queue or topic, based on the value of the MessageID property.

Storage queue messages can duplicate without warning.

Metadata

Service Bus gives you 2 parts of a message header+body. This is a very useful feature for a globally deployed infrastructure. Letting you decorate your messages with things like region name and instance id. Queue messages are simple strings. On the other hand Azure storage queues provide support for arbitrary attributes that can be applied to the queue description, in the form of name/value pairs. So you can decorate the message in Service Bus and decorate the queue with storage queues

Delivery Guarantee

Service Bus offers At-Most-Once and At-Least-Once while Queues only offer At-Least-Once delivery. This could limit our ability to use queues if concurrent subscribers are ever a problem.

Performance

Azure storage queues offer a 10ms latency (within a datacenter) while Service Bus 20-25ms latency. Service bus does offer long-polling which would be even better than 10ms if you have a need for it.

Security

Storage queues use the primary/secondary shared key thing while service bus provides RBAC via Active Directory with Sender/Receiver/Admin roles.

references

• https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx
• http://azure.microsoft.com/en-us/pricing/details/storage/
• http://azure.microsoft.com/en-us/pricing/details/service-bus/
• https://alexandrebrisebois.wordpress.com/2013/10/20/windows-azure-storage-queues-vs-windows-azure-service-bus-queues/

Answer 6

As I understand it, the Service Bus (as was) has had queues for a while, but these are not guaranteed to deliver the message - bon chance!