[ee] fix: ignore arg values in default debounce key#8845
Closed
rubenfiszel wants to merge 1 commit intomainfrom
Closed
[ee] fix: ignore arg values in default debounce key#8845rubenfiszel wants to merge 1 commit intomainfrom
rubenfiszel wants to merge 1 commit intomainfrom
Conversation
The default debounce key baked all non-accumulated arg values into a colon-joined suffix, so any varying non-accumulated field (e.g. a timestamp, a kafka offset, or a msg id present in a preprocessor's output) silently broke debouncing — every call got a unique key and no jobs ever collapsed into a single debounced run. The default key is now just the runnable's fully-qualified path. The surviving job keeps its own (latest) non-accumulated args, and debounce_args_to_accumulate is still merged across the batch at pull time. Users who want per-arg debouncing can still set a custom debounce_key template (e.g. "pp_$args[region]"). Drops the now-unused args_to_ignore_if_default parameter from resolve_debounce_key. Companion: windmill-labs/windmill-ee-private#<pending> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploying windmill with
|
| Latest commit: |
daab7e4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9701764f.windmill.pages.dev |
| Branch Preview URL: | https://kafka-trigger-debounce-fix.windmill.pages.dev |
Contributor
Author
|
Closing for now — deferring the default-key semantics change. The docs PR (windmill-labs/windmilldocs#1353) documenting current behavior stays open. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The bug: Jobs that should debounce into a single run were firing individually whenever the runnable's args carried any per-call variation outside
debounce_args_to_accumulate. This most commonly showed up on Kafka triggers into flows with a preprocessor step + debouncing, where the preprocessor returned anything that differed per message (timestamp, kafka offset, msg id, etc.) — every message got its own job even though the user clearly intended them to collapse.Root cause:
resolve_debounce_keybuilt the default key asOnly the first entry of
debounce_args_to_accumulatewas excluded. Any other arg — including fields the user had no reason to think about, like aDate.now()they returned for logging — made the key unique per call and silently broke debouncing.Fix: The default key is now just the runnable's fully-qualified path:
All calls to the same runnable now land in the same batch regardless of their args. The surviving job still keeps its own (latest) non-accumulated args, and
debounce_args_to_accumulateis still merged across the batch at pull time. Users who want per-arg debouncing can still set an explicitdebounce_keytemplate (e.g."pp_$args[region]").This matches how users expect debouncing to behave — batch everything on this runnable within the delay window, then run once with the accumulated payload.
Changes
backend/windmill-queue/src/jobs.rs:resolve_debounce_keydrops theargs-based suffix in its default branch; the unusedargs_to_ignore_if_defaultparameter is removed. The explicitdebounce_keytemplate path (with$args[...]interpolation) is unchanged.windmill-queue/src/jobs_ee.rs(companion EE PR): updates the two call sites inmaybe_debounceandmaybe_debounce_post_preprocessingto drop the removed parameter, and skips the now-unuseddebounce_args_to_accumulatefield in theirDebouncingSettingsdestructuring.backend/windmill-queue/tests/debounce_test.rs: flipstest_post_preprocessing_args_to_accumulate_different_non_accumulated(renamed to..._non_accumulated_ignored) to assert that jobs with the same runnable but differing non-accumulated args now collapse. All other debounce tests continue to pass unchanged.Semantic change
This is a breaking behavior change for anyone whose flow/script relied on the old "differentiate by non-accumulated args" default. They must now set an explicit
debounce_keytemplate to keep that behavior. The existingtest_post_preprocessing_debounce_args_differentiationtest already demonstrates the template pattern:debounce_key: Some("pp_$args[region]").Test plan
cargo test -p windmill-queue --test debounce_test --features private,enterprise -- --skip stress→ 50 passedauto_commit=false) → flow with preprocessor returning{items:[value], fixed:'const', leaked: Date.now()}→ debounce_delay_s=5, debounce_args_to_accumulate=["items"]. Before the fix: 5 kafka messages → 5 separate jobs. After the fix: 5 kafka messages → 1 surviving job withitems:[v1,v2,v3,v4,v5]and 4 jobs marked"Debounced by ...".test_post_preprocessing_debounce_args_differentiation(which uses an explicitdebounce_keytemplate) still passes, confirming the custom-template path is untouched.Generated with Claude Code
Summary by cubic
Fixes debouncing so calls to the same runnable collapse even when non-accumulated args differ. This resolves Kafka-trigger flows where preprocessor output (timestamps, offsets, IDs) made every job unique.
Bug Fixes
{workspace}/{flow|script}/{path}); arg values are ignored.args_to_ignore_if_defaultfromresolve_debounce_key; explicitdebounce_keytemplates with$args[...]remain supported.Migration
debounce_key(e.g."pp_$args[region]").Written for commit daab7e4. Summary will update on new commits.