perf: wire apply_diff_eq into all filter paths — O(candidates) vs O(base)#205
Merged
perf: wire apply_diff_eq into all filter paths — O(candidates) vs O(base)#205
Conversation
…ase) Replace fused_cow (clones 13MB base bitmap on dirty fields) with apply_diff_eq/union_with_diff across ALL four filter clause paths: 1. Eq: ff.apply_diff_eq(key, acc) — O(candidates) AND 2. In (non-complement): ff.union_with_diff(&in_keys, acc) — single lock 3. In (complement): ff.union_with_diff(&exclude_keys, acc) for subtraction 4. NotEq: ff.apply_diff_eq(key, acc) then subtract 5. NotIn: ff.union_with_diff(&keys, acc) then subtract Before: get_versioned + fused_cow = O(base) per dirty clause, cloning the full base bitmap (13MB at 109M records). At 372ms filter time on broad queries, this is the P99 bottleneck. After: apply_diff_eq computes result at O(candidates) cost using the diff overlay — no base clone needed. Expected 2-10x improvement per dirty clause depending on candidate set size. These methods already existed in FilterField but were never wired into the executor hot path. Bitmap scout P1 recommendation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fused_cow(clones 13MB base on dirty bitmaps) withapply_diff_eq/union_with_diffacross all 5 filter clause paths (Eq, In, In-complement, NotEq, NotIn)FilterFieldbut were never wired into the executor hot path — bitmap scout P1 recommendationProblem
Prod traces show filter evaluation taking 40-372ms on cache misses. The fused_cow path clones the entire 13MB base bitmap whenever a field is dirty (has pending diffs). At 109M records, this is the dominant P99 bottleneck after the doc-read fix (PR #204).
Changes
src/executor.rs: All filter paths intry_and_by_refrewired fromget_versioned + fused_cowtoapply_diff_eq/union_with_diffTest plan
query_filter_secondsP95/P99🤖 Generated with Claude Code