Exceeds the limit (262144 bytes) currently allowed

问题:

We are using SendBatch() method of Azure service bus SDK 3.1.7 to send data to Azure Event Hub. Following is the code snippet :

foreach (var packet in transformedPackets)
        {
            EventData eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(packet, settings)));;
            try
            {
                eventData.Properties.Add(AppConstants.TenantDeploymentUniqueIdKey, tenantDeploymentUniqueId);
                eventData.Properties.Add(AppConstants.DataTypeKey, DataTypeKey);
                byteCount += eventData.SerializedSizeInBytes;                   
                if (byteCount > MaxBatchSize)
                {
                    sender.SendBatch(transformedMessages);
                    transformedMessages.Clear();
                    byteCount = eventData.SerializedSizeInBytes;
                }
                transformedMessages.Add(eventData);
            }
            catch (System.Exception)
            {
                eventData.Dispose();
                throw;
            }

        }

Event after checking the SerializedSizeInBytes property before adding the event data to the batch (Maximum allowed limit : 256 KB), we are getting the following exception :

The received message (delivery-id:0, size:262279 bytes) exceeds the limit (262144 bytes) currently allowed on the link. at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception) at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.MessageSender.RetrySenderEventDataAsyncResult.End(IAsyncResult r) at Microsoft.ServiceBus.Messaging.MessageSender.EndSendEventData(IAsyncResult result) at Microsoft.ServiceBus.Messaging.EventHubSender.SendBatch(IEnumerable1 eventDataList)

回答1:

When events are translated into a single AmqpMessage some additional information is added to the batch header.

Make sure the aggregate size of all EventData falls below 250k.

Here you can find detailed information for this issue: How to use client-side event batching functionality while Sending to Microsoft Azure EventHubs

回答2:

I too was facing the same issue on Event Hub. But smart way to handle these function is to make use of Recursive call. Here by Posting the sample code which working fine for me.

Method 1 : Process the Normal SendBatchAsync Operation

public async Task SendBatchToEHAsync(IEnumerable<string> inputContent)  // Taking List of String to process (50000 Lines Approx)
{
    try
    {
        var batch = new List<EventData>();
        foreach (var item in inputContent) // Add it to Batch
            {
                var bind = new EventData(Encoding.UTF8.GetBytes(item));
                batch.Add(bind);
            }
            try
            {
                await eventHubClient.SendBatchAsync(batch);

            }
            catch (MessageSizeExceededException)
            {
                await ReprocessQuotaExceededBatch(batch);
            }
    }
    catch (Exception ex)
    {

        throw;
    }
}

Method 2: Handle and right the recursive function to handle "MessageSizeExceededException"

 public async Task ReprocessQuotaExceededBatch(List<EventData> batch)
{
    try
    {
        var noOfCalls = 2;
       var noOfRecordsPerSplitBatch = (int)Math.Ceiling((double)batch.Count / noOfCalls);
        var counter = 0;
        while (counter < noOfCalls)
        {
            var insertSubList = GetSubListForEventDataBatch(batch, counter);
            try
            {
                await eventHubClient.SendBatchAsync(insertSubList);

            }
            catch (MessageSizeExceededException)
            {
                await ReprocessQuotaExceededBatch(insertSubList);
                // Perform SPlit processing in case of this exception occures
            }
            counter++;
        }
    }
    catch (Exception ex)
    {

        throw;
    }
}

Method 3: Helper method to make splitting for Items

 private static List<EventData> GetSubListForEventDataBatch(List<EventData> list, int count)
{

    var sList = list.Skip(count * noOfRecordsPerSplitBatch).Take(noOfRecordsPerSplitBatch);
    List<EventData> subList = new List<EventData>();
    subList.AddRange(sList);
    return subList;
}

Hope this solve the Problem for others. Please write down your improvements for this code if any.