Skip to content

Conversation

@john-z-yang
Copy link
Contributor

@john-z-yang john-z-yang commented Dec 16, 2024

Overview

Removes incremental rebalancing logic from the kafka consumer state machine.
We currently have a working implementation here. However, taking advantage of incremental rebalancing requires a little further work:

  • The rdkafka library currently has a bug with committing consumer offset manually during incremental rebalance, causing the consumer to lose its assignment when encountering an illegal generation error. We generally don't use manual offset commit and utilize the auto commit from rdkafka, but we still manually commit offset on shutdown signals. The way the PR above works around this issue is to shutdown the actors, poll the consumer for a predetermined amount of time before actually shutting down. But to use this in production we should wait for the tracking issue to be resolved and see what course of action the rdkfaka team decides to take before trying again.
  • We currently re-create all actors even on partition assignment, we need to just increase actors without shutting down to actually reduce the time spent rebalancing. (This should be a relatively straightforward change).

@john-z-yang john-z-yang force-pushed the john/remove-coop-rebl branch from 27656b8 to fa789d6 Compare December 16, 2024 21:39
@john-z-yang john-z-yang marked this pull request as ready for review December 16, 2024 22:07
@john-z-yang john-z-yang requested a review from a team December 16, 2024 22:07
@john-z-yang john-z-yang merged commit e8cf0f7 into main Dec 16, 2024
6 checks passed
@john-z-yang john-z-yang deleted the john/remove-coop-rebl branch December 16, 2024 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants