Time limit on Azure Webjobs triggered by Queue

Question

Are there any time limits on the queue triggered function inside an azure web job. My function takes about 20-30 mins since it needs to go over a lot of records in the DB. The function status in the end is Never Finished. I put in the settings.job file with { "stopping_wait_time": 1800 } - with no effect. But within five minutes or so its status is set to Never Finished. I searched and yes found some ppl asking similar questions - with no definitive answers. I found some exceptions in the logs and the exceptions seemed to be generated from Azure storage. I searched and found there were some issues with logging, so i commented out logging code within the function - still no go. Also i found one question that mentioned some connection strings - so now i have these; AzureWebJobsStorage, AzureWebJobsDashboard, AzureJobsRuntime, AzureJobsData - all pointing to the same storage account - still no go.

I tried to debug, and see where the timeout occurs and it seems like it is happening even before my function is actually hit,

Output is something like below;

A first chance exception of type 'System.Net.WebException' occurred in System.dll
A first chance exception of type 'System.Net.WebException' occurred in System.dll
A first chance exception of type 'System.TimeoutException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in mscorlib.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in mscorlib.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.Azure.WebJobs.Host.dll
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>PnCProcessor.exe</AppDomain><Exception><ExceptionType>Microsoft.WindowsAzure.Storage.StorageException, Microsoft.WindowsAzure.Storage, Version=4.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35</ExceptionType><Message>The client could not finish the operation within specified timeout.</Message><StackTrace>   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.&amp;lt;&amp;gt;c__DisplayClass1`1.&amp;lt;CreateCallback&amp;gt;b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.&amp;lt;ExecuteAsync&amp;gt;d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.&amp;lt;RunAsync&amp;gt;d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()</StackTrace><ExceptionString>Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---&amp;gt; System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.&amp;lt;&amp;gt;c__DisplayClass1`1.&amp;lt;CreateCallback&amp;gt;b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.&amp;lt;ExecuteAsync&amp;gt;d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.&amp;lt;RunAsync&amp;gt;d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Request Information
RequestID:
RequestDate:
StatusMessage:
</ExceptionString><InnerException><ExceptionType>System.TimeoutException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The client could not finish the operation within specified timeout.</Message><StackTrace>   at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

System.TimeoutException: The client could not finish the operation within specified timeout.

I am using a S1 web app and a standard-GRS storage account for this.

So questions are;
1. Are there any time limitations on the time the queue triggered function takes?
2. If there are time limitations is there any way to override those?
3. I have to use a queue triggered function since i need to pass some parameters to the function - which are passed from another scheduled web job. I can make an on-demand webjob and pass the parameters on the command line - but then is there a way to call an on-demand webjob from a scheduled webjob?

Thanks for your help!

Answer 1

No there is no limit on execution time of queue triggered job functions. However there are a few things you have to ensure you set up properly so your JobHost continues to run. When using the Azure WebJobs SDK you need to deploy your JobHost Console exe as a continuously running WebJob, and enable Always On. Both of those are described in detail on this documentation page under the "Create a continuously running WebJob" section.

See the "Notes" section of that page for relevant details on the timeout limits when running in the Free tier (20 minute timeout), which can be removed by moving to Basic or Standard tier.