feat(zql): countdown-based sampling for MeasurePushOperator#5603
Draft
Karavil wants to merge 4 commits intorocicorp:mainfrom
Draft
feat(zql): countdown-based sampling for MeasurePushOperator#5603Karavil wants to merge 4 commits intorocicorp:mainfrom
Karavil wants to merge 4 commits intorocicorp:mainfrom
Conversation
|
Someone is attempting to deploy a commit to the Rocicorp Team on Vercel. A member of the Team first needs to authorize it. |
Adds configurable sampling to MeasurePushOperator to reduce per-push overhead from performance.now() + tdigest stats computation. Sampling is controlled via metricsDelegate options: - disableMetrics: true skips all measurement - metricsSampleRate: 0-1 controls sampling frequency Default behavior is unchanged (sample rate = 1, measure every push). * Add resolveSampleEvery() to convert delegate options to sample interval * Add #sampleEvery/#sampleCountdown fields for countdown-based sampling * Skip performance.now() + addMetric() calls for non-sampled pushes
26f2390 to
7eafa8c
Compare
added 2 commits
February 24, 2026 20:51
Covers: default sampleRate=1 (every push), disableMetrics=true, sampleRate=0, fractional rates (0.5→every 2nd, 0.25→every 4th), clamping for out-of-range values (<0, >1).
41k pushes across four configurations: sampleRate=1 (default), disableMetrics=true, sampleRate=0.01, and sampleRate=0 (disabled).
Add benchmark cases at 2K, 15K, and 40K push counts to show how sampling overhead scales with poke size. * Replace single 41K test with parameterized loop over [2K, 15K, 40K] * Each scale tests all four variants: sampleRate=1, sampleRate=0.01, disableMetrics=true, sampleRate=0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MeasurePushOperator.push()callsperformance.now()and feeds every result into a TDigest on every single push. During large pokes (41K+ diffs from initial sync or batch mutations), this per-push measurement path adds measurable overhead. At 40K pushes, the measurement logic alone accounts for roughly 3x throughput reduction compared to skipping it entirely.Solution
Countdown-based sampling: measure every Nth push instead of every push.
resolveSampleEvery()computes the interval once at construction from ametricsSampleRateoption, converting the rate into an integer countdown. The hot path is a single integer decrement and compare. No per-pushMath.random(), no per-push branching beyond the countdown check.Two new optional properties on the
metricsDelegateobject:disableMetrics: true: skip all measurement. The push method forwards directly to the output with zero instrumentation overhead.metricsSampleRate: number: fraction of pushes to measure.1= every push (current default, no behavior change).0.01= every 100th push.0= disabled. Clamped to [0, 1], converted to interval viaMath.round(1 / rate).Benchmark results
Measured with vitest bench. Each row is the mean time per iteration (lower is better). "Overhead saved" compares
sampleRate=0.01against thesampleRate=1baseline.2,000 pushes
15,000 pushes
40,000 pushes
The overhead ratio is consistent across scales:
sampleRate=0.01recovers roughly 60% of the measurement cost while still collecting statistically useful digest data. At 40K pushes, that is ~1.9ms saved per poke.Backward compatibility
Default
sampleRateis 1, which means every push is measured. Existing consumers that do not passdisableMetricsormetricsSampleRatesee identical behavior. TheMetricsDelegateinterface is unchanged. The new properties are read from the delegate object at construction time via duck typing, so no type changes propagate to callers.Test plan
MeasurePushOperatortests pass unchanged (no regressions in default behavior)packages/zql/src/query/measure-push-operator.bench.tsvalidates overhead reduction at 2K, 15K, and 40K push counts