How do I recover from UnroutableError after RabbitMQ is restarted?

[alanbchristie]

When I restart RabbitMQ otehr Pods in my cluster (specifically Python Pods) that had connected to the service encounter NO ROUTE errors. No matter what I do I do not seem to be able to recover from the problem without restarting the client (Python) Pod. But, surely, I should be able to recover from the problem without restarting the client Pod?

I've used pika and the official rabbit ampq driver, both behave the same. In the discussion there (pika/pika#1543) the concern was why I don't get a connection failure: -

1. I establish a BlockingConnection connection successfully.
2. I force RabbitMQ to restart (kill the Pod)
3. The connection fails with the expected UnroutableError
4. Any attempt to close and re-establish the connection fails with continual UnroutableError errors

[lukebakken]

Hi @alanbchristie

Both pika/pika#1543 and this issue are missing important information - specifically, the RabbitMQ and Erlang version you're using, as well as details about your RabbitMQ topology and application code

Again, your symptoms point to something specific in your application code or environment. What happens when you run your application against a single-node RabbitMQ running outside of k8s? I suspect you don't see this issue.

Here are some assumptions I've made (please correct them if wrong):

• You are using the latest RabbitMQ docker image in your k8s environment.
• Your pika application is publishing to a quorum queue.
• Your pika application has publisher confirmations enabled (i.e. confirm_delivery has been called). I say this because pika will only raise UnroutableError in this case:

          :raises UnroutableError: raised when a message published in
          publisher-acknowledgments mode (see
          `BlockingChannel.confirm_delivery`) is returned via `Basic.Return`
          followed by `Basic.Ack`.