fix: route group coordination requests to a single broker to prevent split brain#125
Merged
novatechflow merged 1 commit intoKafScale:mainfrom Mar 2, 2026
Merged
Conversation
…split brain Closes KafScale#121. Brokers now acquire an etcd lease before coordinating any group operation. Only the lease holder can coordinate — others reject with NOT_COORDINATOR and the proxy retries on the correct broker. Also fixes: silent error swallowing on transient etcd failures, wrong byte-offset NOT_COORDINATOR detection on flexible protocol versions, false-positive byte scanning, DescribeGroups multi-group retry loop, and connect-failure retry abort in the proxy.
novatechflow
approved these changes
Mar 2, 2026
Collaborator
novatechflow
left a comment
There was a problem hiding this comment.
Nice one! Thanks :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #121.
When multiple consumers in the same group connect through different brokers, each broker independently tries to coordinate the group. This causes split brain: both brokers assign partitions, neither knows about the other's assignments, and messages get processed multiple times with no errors or warnings.
This PR fixes that by making brokers acquire an etcd lease before coordinating a group. Only the lease holder can coordinate — all other brokers reject the request with
NOT_COORDINATOR, and the proxy retries on the correct broker.What changed
Broker — Before handling any group operation (join, sync, heartbeat, leave, offset commit/fetch, describe), the broker tries to acquire an etcd lease for that group. If another broker already holds it, the request is rejected with
NOT_COORDINATOR. If etcd is unreachable, the request is rejected withREQUEST_TIMED_OUTinstead of silently proceeding (which was the old behavior that allowed two brokers to coordinate the same group simultaneously).Proxy — The proxy watches etcd for group lease ownership and routes group requests to the owning broker. If the broker responds with
NOT_COORDINATOR, the proxy invalidates its cache and retries on a different broker. The old detection logic used hard-coded byte offsets that broke on newer protocol versions and produced false positives on normal data. This is replaced with proper response parsing. Multi-group DescribeGroups requests are forwarded once without retry since different groups may live on different brokers — the client handles per-group errors natively.Lease manager — The partition and group lease managers shared identical logic for session management, acquire/release, and etcd transactions. Both now delegate to a shared generic
LeaseManagerwith thin type-safe wrappers on top.Preexisting bugs fixed along the way
REQUEST_TIMED_OUT.NOT_COORDINATORon newer protocol versions (flexible headers shifted byte offsets). Replaced with proper response parsing.