How do I delay the execution of an Azure function

Question

I am using an Azure Function (written in C#) that is triggered when there is a new message on the storage queue. The function is calling an external web-service to process the message, and when the external service is responding, everything is OK.

The problem is when the external web-site is down, and the web-client gives an exception (without a try/catch block around it). What happens then is that the function is retried (by azure) 5 times, with less than a second between each attempt. As you can imagine, the web service will probably still be down for all 5 attempts which will make azure move the message to the "poison"-queue.

Is it possible to configure the the amount of time it waits before retrying, or do i need to set up another azure function that runs once a minute to check the poison-queue for messages that then needs to be recreated in the normal queue?

Answer 1

We don't currently expose a way for you to configure the visibility timeout for failed messages. It is currently hardcoded to TimeSpan.Zero as you've noticed. I agree with you that we should allow this to be customized. I've logged an issue in our public repo with more details here.

Answer 2

We ended up handling the visibilityTimeout parameter "manually": we do not use output bindings but schedule the message in the queue for later processing using the Azure storage SDK.

See https://github.com/teamdigitale/digital-citizenship-functions/blob/master/lib/utils/azure_queues.ts#L86

 queueService.updateMessage(
        queueName,
        queueMessage.id,
        queueMessage.popReceipt,
        visibilityTimeoutSec,
        err => {
          context.done( err );
        }
      );

This let us plug an exponential backoff strategy for reties without putting the process to sleep.

Answer 3

The function is calling an external web-service to process the message, and when the external service is responding, everything is OK.

Why not signal the service status with a queue message.

Function is triggered
        \  /
         \/
Do we have a message in service-status queue?
        \  /
         \/    
Wait 30 seconds before making web-service call / else call it right away.
        \  /
         \/
Can't reach web-service? -> Drop message in service-status queue.

If you have multiple instances of that Function App you'll need to signal with either a durable file on storage or an entry in Table Storage.