I am using an Azure Function (written in C#) that is triggered when there is a new message on the storage queue. The function is calling an external web-service to process the message, and when the external service is responding, everything is OK.
The problem is when the external web-site is down, and the web-client gives an exception (without a try/catch block around it). What happens then is that the function is retried (by azure) 5 times, with less than a second between each attempt. As you can imagine, the web service will probably still be down for all 5 attempts which will make azure move the message to the "poison"-queue.
Is it possible to configure the the amount of time it waits before retrying, or do i need to set up another azure function that runs once a minute to check the poison-queue for messages that then needs to be recreated in the normal queue?
We don't currently expose a way for you to configure the visibility timeout for failed messages. It is currently hardcoded to TimeSpan.Zero
as you've noticed. I agree with you that we should allow this to be customized. I've logged an issue in our public repo with more details here.
The function is calling an external web-service to process the message, and when the external service is responding, everything is OK.
Why not signal the service status with a queue message.
Function is triggered
\ /
\/
Do we have a message in service-status queue?
\ /
\/
Wait 30 seconds before making web-service call / else call it right away.
\ /
\/
Can't reach web-service? -> Drop message in service-status queue.
If you have multiple instances of that Function App you'll need to signal with either a durable file on storage or an entry in Table Storage.
We ended up handling the visibilityTimeout parameter "manually": we do not use output bindings but schedule the message in the queue for later processing using the Azure storage SDK.
See https://github.com/teamdigitale/digital-citizenship-functions/blob/master/lib/utils/azure_queues.ts#L86
queueService.updateMessage(
queueName,
queueMessage.id,
queueMessage.popReceipt,
visibilityTimeoutSec,
err => {
context.done( err );
}
);
This let us plug an exponential backoff strategy for reties without putting the process to sleep.