I have a long running process which I call in my S

Question

I have a long running process which performs matches between millions of records I call this code using a Service Bus, However when my process passes the 5 minute limit Azure starts processing the already processed records from the start again.

How can I avoid this

Here is my code:

private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
 {
   long receivedMessageTrasactionId = 0;
   try
   {
     IQueueClient queueClient = new QueueClient(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);

     // Process the message
     receivedMessageTrasactionId = Convert.ToInt64(Encoding.UTF8.GetString(message.Body));

     // My Very Long Running Method
     await DataCleanse.PerformDataCleanse(receivedMessageTrasactionId);
            //Get Transaction and Metric details

     await queueClient.CompleteAsync(message.SystemProperties.LockToken);
   }
   catch (Exception ex)
   {
     Log4NetErrorLogger(ex);
     throw ex;
   }
}

Answer 1

Messages are intended for notifications and not long running processing.

You've got a fewoptions:

1. Receive the message and rely on receiver's RenewLock() operation to extend the lock.
2. Use user-callback API and specify maximum processing time, if known, via MessageHandlerOptions.MaxAutoRenewDuration setting to auto-renew message's lock.
3. Record the processing started but do not complete the incoming message. Rather leverage message deferral feature, sending yourself a new delayed message with the reference to the deferred message SequenceNumber. This will allow you to periodically receive a "reminder" message to see if the work is finished. If it is, complete the deferred message by its SequenceNumber. Otherise, complete the "reminder" message along with sending a new one. This approach would require some level of your architecture redesign.
4. Similar to option 3, but offload processing to an external process that will report the status later. There are frameworks that can help you with that. MassTransit or NServiceBus. The latter has a sample you can download and play with.

Note that option 1 and 2 are not guaranteed as those are client-side initiated operations.