Sending 1000 brokered messages to the service bus

Question

I have an application wherein data is fetched from the SQL DB and sent to the service bus as brokered message. These are the steps:

1. Data fetched from the DB(in batches of 1000)
2. Each row of data converted into Brokered Message and added into a list.
3. The list of 1000 brokered messages is sent to the service bus using SendBatchAsync method.

It is at the 3rd step that I am facing the issue. This is the code for that:

public async Task SendMessagesAsync(List<BrokeredMessage> brokeredMessageList)
        {
            try
            {
                var topicClient = CreateTopicClient();
                await topicClient.SendBatchAsync(brokeredMessageList);
            }
            catch(Exception ex)
            {
                throw ex;
            }
        }

when the compiler comes to SendBatchAsync method, it gives an error that Error during communication with Service Bus. Check the connection information, then retry. with the inner exception being:

Internal Server Error: The server did not provide a meaningful reply; this might be caused by a premature session shutdown. TrackingId:some guid here

However if I try sending 100 messages, it works fine. What can I do to make it send 1000 messages at a time?

Note: each message size is 1445 bytes

Answer 1

You can use a premium namespace in Azure Service Bus. This allows messages upto 1MB. Though the header is still limited to 64K.

PS - Check the pricing before using the premium namespace.

For more information ->

https://blogs.msdn.microsoft.com/servicebus/2016/07/07/things-to-know-about-premium-messaging/

Answer 2

Unfortunately you can't because your total payload size is about 1.4 MB (1445 bytes * 1000) whereas maximum size of the batch allowed is 256 KB.

Ref: https://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.topicclient.sendbatch.aspx (Remarks section)

The maximum size of the batch is the same as the maximum size of a single message (currently 256 Kb).

I guess you would need to split the batch further into smaller batches so that you don't exceed 256K limit.