the shovel deletes itself prematurely before all the messages in the source queue are transferred

[declan-zZ]

Describe the bug

when creating a shovel with below settings:

• src-delete-after = queue-length
• src-prefetch-count = 1000
• ack-mode=on-confirms

The shovel deletes itself prematurely before all the messages in the source queue are transferred.

Reproduction steps

1. create a source queue and prepare 20k msgs there
2. create a dynamic shovel with settings as described above

Expected behavior

The shovel deletes itself only after all the messages in the source queue are transferred.

Additional context

If I update src-prefetch-count to a smaller count (say 1), the shovel can work as expected.

[mkuratczyk]

1. You did't provide the RabbitMQ version.
2. Please provide the actual commands used to reproduce - without that we need to guess some of the parameters you are using and we might guess wrong
3. ideally, please test on the latest 4.2.0 beta version - there are many shovel improvements there

[michaelklishin]

@declan-zZ the goal behind the auto-deleted shovels was never to consume a "slice" of a queue backlog.

It's always been to let you transfer everything in a queue and then let the shovel stop. But because "everything" is hard to define in the world of messages because the data set is dynamic, the setting says "N messages". So it exists for moving data from a source that's no longer needed but still has some data, without writing a small app/script to move the messages for you.

We will not guess as to what RabbitMQ version you use, whether it is [still covered by community support](https://www.rabbitmq.com/release-information and so on.

[declan-zZ]

hi @michaelklishin @mkuratczyk
I'm using 4.1.2 for the testing. sorry for missing such key info in the question.
In my test, the data set in the source queue was static, I stopped the publisher and then started the auto-delete shovel.

Any chance the shovel doesn't wait for the ack from the "re-publisher" and terminiates itself as no more msgs in the source queue? However, some msgs could still be in-the-fly and not yet acked. Once the shovel/consumer is terminated, msgs in-the-fly are requeued/left-over. That explains the issue unlikely appears if the prefetch is smaller.

[gomoripeti]

what protocol do you use on the source and dest side? AMQP 0.9.1 or 1.0? Direct or via the network? It would be best if you copy or screenshot the complete definition of the shovel as there are still missing information.

[declan-zZ]

what protocol do you use on the source and dest side? AMQP 0.9.1 or 1.0? Direct or via the network? It would be best if you copy or screenshot the complete definition of the shovel as there are still missing information.

It's AMQP 0.9.1 and direct. I'm trying shovel messages (message size is 4kb) between different vhosts within the same Rabbit cluster.

when the shovel is running:

when the shovel is stopped/auto-deleted:

[dcorbacho]

@declan-zZ Thanks for your report. I haven't seen it happening and I've worked recently a lot with auto-delete shovels, but it can happen as shovel doesn't wait for message confirmation when it auto-deletes. It just counts messages sent. So if your app takes some time to acknowledge messages, the shovel can stop before.

Seeing this side effect, I would consider it a bug. I'll look into it.

[lukebakken]

@declan-zZ I created the following repository to make it easy to try and reproduce what you report:

https://github.com/lukebakken/rabbitmq-server-14639

It works correctly every time. Please note how I declare the dynamic shovel:

https://github.com/lukebakken/rabbitmq-server-14639/blob/main/Makefile#L25

Parameters:

{"src-protocol": "amqp091", "src-uri": "amqp://rmq/vhost0", "src-queue": "rmq-14639",
"dest-protocol": "amqp091", "dest-uri": "amqp://rmq/vhost1", "dest-queue": "rmq-14639",
"ack-mode": "on-confirm", "src-prefetch-count": 1000, "src-delete-after": "queue-length"}

If you really are using "ack-mode": "on-confirms" (with an s), that is not the correct parameter.

@declan-zZ in the future, when you report an issue to any software project, put in the effort to make it simple and easy for the maintainers to reproduce what you report. It will be appreciated.

[michaelklishin]

@lukebakken thank you. @mkuratczyk @dcorbacho and I are investigating.

[lukebakken]

Can you reproduce the issue? I'm curious!

[michaelklishin]

@lukebakken AFAIK Diana can. It just takes a large enough N (the number of messages to transfer), in at least tens of thousands.

[dcorbacho]

I can't reproduce, but I understand how it can happen. I'll get on with this shortly