We have observed the following behavior of RabbitMQ and are trying to understand if it is correct and how to resolve it.
Scenario:
- A (persistent) message is delivered into a durable queue
- The (single) Consumer (Spring-AMQP) takes the message and starts processing => Message goes from READY to UNACK
- Now the broker is shut down => Client correctly reports "Channel shutdown"
- The consumer finishes the processing, but can not acknowledge the message as the broker is still down
- Broker is started again => Client reconnects
As a result, one message remains unack'ed forever (or until the client is restarted).
Side note: In the Rabbit Admin UI, I can see that two channels are existing now. The "dead" one that was created before the broker restart, containing the unacked message and a new one that is healthy.
Is this behavior expected to be like that? It seems to me "correct" in the way, that RabbitMQ can not know after the broker restart, whether the message processing was completed or not. But what solution would exist than to get that unacked message back into the queue and to heal the system without restarting the consumer process?