How do I increase the timespan of the TimeoutPersisterReceiver? it currently retries for ~1min after arming before firing. I have looked in the documentation but cannot see a config setting for it.
We have been having some network issues that prevented a service from talking to the database causing the RepeatedFailuresOverTimeCircuitBreaker to arm itself.
INFO NServiceBus.CircuitBreakers.RepeatedFailuresOverTimeCircuitBreaker [(null)] - The circuit breaker for TimeoutStorageConnectivity is now in the armed state
The database will eventually come back online but by the time it does the circuit breaker has been fired and the Bus has been disposed. I need to increase the time span of the NServiceBus.Timeout.Hosting.Windows.TimeoutPersisterReceiver.
NServiceBus.CircuitBreakers.RepeatedFailuresOverTimeCircuitBreaker [(null)] - The circuit breaker for TimeoutStorageConnectivity will now be triggered
FATAL NServiceBus [(null)] - Repeated failures when fetching timeouts from storage, endpoint will be terminated.
You can try to configure it in a class that implements IWantToRunWhenBusStartsAndStops and use the SecondsToSleepBetweenPolls property of the TimeoutPersisterReceiver.
However, I would add that public access to this configuration option was removed in later versions, and I'm not 100% sure why. There may be dragons. See http://bit.ly/1ToyOHb (GitHub).
Circuit breaker behavior
After researching the code it seems that you would be able to configure the timeout poll interval via TimeoutPersisterReceiver.SecondsToSleepBetweenPolls but that does not really matter.
The embedded circuit breaker configuration cannot be adjusted. If an error occurs it is armed and if no successful poll succeeds within 2 minutes the circuit breaker action will be triggered which is calling raising a critical error.
Increasing the poll interval will not help here. If you would make the poll interval more than 2 minutes the circuit breaker will always be armed thus trigger the action.
https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/Timeout/Hosting/Windows/TimeoutPersisterReceiver.cs#L150
https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/CircuitBreakers/RepeatedFailuresOverTimeCircuitBreaker.cs
There is nothing that you can do unfortunately except building your own version that removes the circuit breaker completely.
This behavior can result in issues when your database server does a fail-over.
Critical error handling
The only possible solution to tackle this condition is by hooking into the critical error handler. Then you know such issue occurs and can do a application shutdown as I suspect that you are currently self-hosting.
You can also try to create a whole new bus instance and 'swap' the broken one with the fresh one but that is a pretty daunting task and I would recommend to just exit your application.
