Azure Web Job-The remote server returned 404

Question

I have a continuos running web job which listens to a start queue and a stop queue with their respective functions. There are 3 predefined process in the web job and the start queue contains a message which mentions the process name to be started while the stop queue contains a message which mentions the process name to be stopped.

When ever the webjob stops a process it crashes and restarts with the above exception since last 4 weeks.

Executing: 'Functions.StartProcess' because New queue message detected on 'start'.
Updated the status of Process A to 1
Process A Started
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Executing: 'Functions.StopProcess' because New queue message detected on 'stop'.
Process A stopped
Disposed Process A
Updated the status of Process A to 3

Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlobSharedImpl.<DeleteBlobImpl>b__1b(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndDelete(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Protocols.PersistentQueueWriter`1.<DeleteAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<DeleteLogFunctionStartedAsync>d__e.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueTriggerExecutor.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ProcessMessageAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The webjob does not even contain any code to delete a blob file.

While debugging, the debugger does not point to any specific line while throwing the exception and hence I have no clue as to what is the actual cause.

PS : It used to run perfectly prior to that and every function's body is enclosed in try catch block. The catch block does not rethrow the exception but just logs the stacktrace and continues.

EDIT

I had to check if the below containers were there

in the webjob's storage and also updated the Microsoft.Web.WebJobs.Publish nuget package. I am really not sure if this fixes the problem for sure though.

Answer 1

I was getting this same error.

What worked for me was updating the WindowsAzure.Storage package in my case from 4.3.0 to 7.1.2, but I would go with latest or whatever your other WebJobs are using (if they are running without issue).

Answer 2

It looks like you were right, you do in fact need to manually create the blob containers, or publish the web job, in which case it looks like the environment will create the containers for you.

When I tried it out initially, I did the publish first, and it worked like a charm. Second project, I am doing it differently, so I ran the console app first, and it kept failing with 404.

So, you need to create:

• azure-webjobs-dashboard
• azure-webjobs-hosts
• azure-jobs-hosts-archive
• azure-jobs-hosts-output

Hope it helps someone.