Skip to content

Helix gateway service for Helix 2.0 #3089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
junkaixue merged 37 commits into from
Dec 9, 2025
Merged

Helix gateway service for Helix 2.0 #3089

junkaixue merged 37 commits into from
Dec 9, 2025

Conversation

@junkaixue
Copy link
Contributor

@junkaixue junkaixue commented Dec 9, 2025

Issues

  • My PR addresses the following Helix issues and references them in the PR description:

resolves Apache Helix 2.0

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Apache Helix Gateway Service

Tests

  • The following tests are written for this issue:

  • The following is the result of the "mvn test" command on the appropriate module:

Changes that Break Backward Compatibility (Optional)

  • My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

new module backward compatible.

Documentation (Optional)

  • In case of new functionality, my PR adds documentation in the following wiki page:

Apache Helix 2.0

Commits

  • My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Code Quality

  • My diff has been formatted using helix-style.xml
    (helix-style-intellij.xml if IntelliJ IDE is used)

junkaixue and others added 30 commits December 9, 2025 10:12
@junkaixue
Set up helix-gateway folder, add PoC code and add protobuf (#2826)
87f0e34 
* Set up helix-gateway folder structure and add PoC code

* Add basic proto for gRPC calls
@xyuanlu @junkaixue
Add protobuff definition and an empty grpc service (#2834)
30b9094 
Add protobuff definition and an empty grpc service
@junkaixue
Build HelixGateway integration test base (#2842)
9449a04 
This test base contains all the setups including:
1. Mock participants which hybrid with gateway services.
2. Controller with start/stop.
3. Resource creation.
4. Only Online/Offline state model.
5. Gateway services with start/stop (now leave empty until gateway service is implemented).

Also start a dummy test to extend test base to make sure all set ups are ready except gateway services.
@xyuanlu @junkaixue
Refactor and remove mock classes (#2841)
2b44e87 
Refactor and remove mock classes
@xyuanlu @junkaixue
Gateway service - service structure dummy class (#2840)
2d619df 
Gateway service - service structure dummy class
@xyuanlu @junkaixue
Gateway util for per key lock and per key blocking queue executor(#2847)
02e6e83 
Gateway util for per key lock and per key blocking queue executor
@xyuanlu @junkaixue
Implement GatewayServiceManager (#2844)
00db566 
Implement GatewayServiceManager
@xyuanlu @junkaixue
Add license to gateway service (#2851)
430d20a 
Add license to gateway service
@zpinto @junkaixue
Implement Helix ST handling logic and HelixGatewayParticipant (#2845)
7c06069 
- Add ST handling logic to support multi top state state model definitions without intermediary states
- Encapsulate the participant manager create, connect, disconnect logic in HelixGatewayParticipant
- Add replica state tracking in HelixGatewayParticipant
@xyuanlu @junkaixue
API to close grpc client stream connection from server side (#2856)
60845e8 
API to close grpc client stream connection from server side
@zpinto @junkaixue
Implement helix manager disconnect and client disconnect handling for…
b37fa83 
… HelixGatewayParticipant.(#2868)

* Implement helix manager disconnect and client disconnect handling for HelixGatewayParticipant.
@xyuanlu @junkaixue
Interfaces of gateway service (#2871)
765398a 
Interfaces of gateway service
@xyuanlu @junkaixue
Refine gateway service interface (#2875)
b31f2b9 
Refine gateway service interface
@xyuanlu @junkaixue
Create Gateway service channel factory (#2883)
c5eaa8a 
This PR introduces a new configuration system for the Helix Gateway service, allowing for more flexible and customizable channel configurations. The main changes include:

-Introduction of GatewayServiceChannelConfig class to manage various channel configurations.
-Implementation of a factory pattern (HelixGatewayServiceChannelFactory) for creating appropriate service channels based on the configuration.
-The PR also includes various improvements in error handling, logging, and code organization.
@xyuanlu @junkaixue
Gateway - User report their shards' current state instead of state tr…
c9f7d05 
…ansition message (#2892)

Gateway - User report their shards' current state instead of state transition message
@zpinto @junkaixue
Add helix-gateway to bump-up.sh and bump-snapshot.sh (#2899)
ad33675 
Add helix-gateway to bump-up.sh and bump-snapshot.sh
@xyuanlu @junkaixue
Gateway - Add GatewayCurrentStateCache for gateway service (#2895)
bbb031a 
This pull request introduces a caching mechanism for the Helix Gateway service. The main changes include:

Addition of a new GatewayCurrentStateCache class in GatewayCurrentStateCache.java, which manages caching of current and target states for instances in a cluster.
@xyuanlu @junkaixue
Gateway - Implementing poll-mode channel (#2900)
d8d1c29 
This pull request introduces significant enhancements to the Helix Gateway service, primarily focusing on implementing a poll-mode channel for communication. The main changes include:

Addition of a new HelixGatewayServicePollModeChannel class that implements the poll-mode communication channel.
Updates to GatewayServiceChannelConfig to support new configuration options for poll-mode.
@frankmu @junkaixue
Create condition based rebalancer (#2846)
0059136 
Create condition based rebalancer
@frankmu @tengfmu @junkaixue
Add missing license for rebalance condition files (#2853)
2663786 
Add missing license for rebalance condition files
Co-authored-by: Tengfei Mu <tmu@linkedin.com>
@frankmu @junkaixue
Helix stickiness rebalancer (#2878)
1cdd250 
Create sticky assignment rebalance strategy
@frankmu @junkaixue
Move existing assignments usage calculation to pre-process stage (#2888)
5a611cc 
Move existing assignments usage calculation to pre-process stage
@frankmu @junkaixue
Add test to ensure no partition movement when nodes restart (#2896)
ba8c9f1 
Add test to ensure no partition movement when nodes restart
@tengfmu @junkaixue
Add null check for getClusterConfig()
c515551
@zpinto @junkaixue
Switch to compatible grpc version (#2904)
38ba820 
* Switch to compatible grpc version.

* Bump snapshot

* Add helix-gateway-service branch to PR CI.
@zpinto @junkaixue
Use grpc.version for io.grpc version (#2905)
9b03e1b
@zpinto @junkaixue
Expose setting gateway service channel to allow external managment of…
6ffa3d3 
… the lifecycle of the channel. (#2913)

Expose setting gateway service channel to allow external managment of the lifecycle of the channel.
@xyuanlu @junkaixue
Gateway - gateway participant update target state in cache (#2910)
663c9aa 
Gateway - gateway participant update target state in cache
@zpinto @junkaixue
Include java stubs in jdk11 jar. (#2920)
f6cecbd 
Add helix-gateway java stubs to for jdk11 jar
@zpinto @junkaixue
Add helix-gateway stubs to exported packages.(#2923)
bd80ae0
xyuanlu and others added 7 commits December 9, 2025 10:12
@xyuanlu @junkaixue
Add an end to end test for helix gateway (#2922)
ef0175d 
Add an end to end test for helix gateway
@zpinto @junkaixue
Synchronize calls to StreamObserver methods (#2934)
3fea937 
- Synchronized calls to onNext, onError, and onComplete to prevent grpc failures
- Ensured thread safety for StreamObserver method invocations
@xyuanlu @junkaixue
Fix gateway e2e test - set timestamp in second instead of ms (#2942)
a685f41
@xyuanlu @junkaixue
Add getter for all target state - gateway service (#2943)
7c4e4dc 
Add getter for all target state - gateway service
@xyuanlu @junkaixue
Add log and refine code style for Gateway service (#2950)
1c55b73
@frankmu @tengfmu @junkaixue
Make StickyRebalanceStrategy topology aware (#2944)
3873e5d 
* Make StickyRebalanceStrategy topology aware

* move canAdd() to a separate function for extendability

* add todo

* address comment and add test

* constructor change for CapacityNode

---------

Co-authored-by: Tengfei Mu <tmu@linkedin.com>
@junkaixue
Fix version in pom
8926f97
@junkaixue junkaixue merged commit 03049ed into master Dec 9, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@junkaixue @xyuanlu @zpinto @frankmu @tengfmu