Race condition on JMS listener scaling down with ActiveMQ Artemis

[fsgonz]

After this changes related to this issue, we are experiencing race conditons when jmsListeners are scaling down and some messages are dropped.

I've seen that this idleReceivesPerTaskLimit was changed which previously was always < 0 as default.
Now, by default it will always eventually result in the consumers to be stopped resulting for this condition.

Should this condition which now is always false by default and previously was always true, be changed to something like this:

if (messageLimit < 0 && (!surplus || idleTaskExecutionCount< idleLimit))\

cc @jhoeller

[jhoeller]

@fsgonz could you elaborate a bit on your scenario: How are messages getting dropped when scaling down? We got two code paths there, and this change in 6.2 was simply meant to use the idle-receives code path for surplus consumers between core and max by default. The latter code path should not result in a message getting dropped; it'd be great to find out more about your case there.

You could try to explicitly set idleReceivesPerTaskLimit to -1 to restore the previous behavior. We should document that option, actually, since the javadoc does not explicitly mention it.

[fsgonz]

@jhoeller yes, actually we've set idleReceivesPerTaskLimit as a WA.
The drop is when one of the consumers already consumed one message and it is stopped just then (because it considers that it was idle). I'm not quite sure the change is responsible for the drop but probably the possibility of the drop was there but as this was by default always negative, it didn't appear till the change was added to effectively do the scale down in those cases.
I will try to create a test and create snapshots with a debugger to explicit the race condition.

[jhoeller]

@fsgonz picking this up again: Could you clarify what kind of "dropped" you are actually experiencing? Are those messages already loaded into the given Session but then not consumed from there? Or are those actually lost in the sense of not getting redelivered in another Session either? What's the acknowledge mode you are using, or are you running in transacted mode?

[fsgonz]

Hi! The messages that are lost are the ones already loaded but they are not redelivered for consumed in another session. Sorry I couldn't get to replicate this. I am able to replicate it through when artemis manages the jms consumers but not with activemq so maybe it is a problem with artemis stopping the consumers.
I'll come back when trying to replicate in case it is spring boot.
For the moment the concern is only for the moment a change in the default behavior but in case it is not a concern for you please feel free to close it and I'll come back after debugging a bit more how the artemis consumers work when been stopped with messages already fetched.