Skip to content

chore: fix sampled field semantics on JudgeResult#137

Merged
jsonbailey merged 3 commits intomainfrom
jb/aic-2200/fix-sampled-semantics
Apr 16, 2026
Merged

chore: fix sampled field semantics on JudgeResult#137
jsonbailey merged 3 commits intomainfrom
jb/aic-2200/fix-sampled-semantics

Conversation

@jsonbailey
Copy link
Copy Markdown
Contributor

@jsonbailey jsonbailey commented Apr 16, 2026

Summary

  • Inverts the sampled field semantics on JudgeResult: sampled=True now means the evaluation was sampled and run; sampled=False means it was not sampled (bypassed by the sampling rate).
  • Previously sampled=True indicated the evaluation was skipped, which was counterintuitive and inconsistent with the field name.
  • Adds an early-return guard in LDAIConfigTracker.track_judge_result() to skip tracking when not result.sampled.

Test plan

  • Updated existing sampling-rate test to assert sampled=False on skip path
  • Added sampled=True assertions to both successful evaluation tests
  • All 114 tests pass

🤖 Generated with Claude Code


Note

Medium Risk
Changes a public result-field meaning and alters when judge metrics are emitted, which can affect downstream consumers and analytics even though the code changes are small.

Overview
JudgeResult.sampled semantics are inverted so sampled=True now indicates the judge evaluation ran (and remains False when skipped by sampling_rate). Judge.evaluate() sets sampled=True only on the run path, and LDAIConfigTracker.track_judge_result() now early-returns when not sampled to avoid emitting metrics for skipped evaluations.

Tests are updated to assert the new behavior for both success and sampling-skip cases.

Reviewed by Cursor Bugbot for commit c4e949e. Bugbot is set up for automated code reviews on this repo. Configure here.

jsonbailey and others added 3 commits April 16, 2026 09:32
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jsonbailey jsonbailey marked this pull request as ready for review April 16, 2026 15:20
@jsonbailey jsonbailey requested a review from a team as a code owner April 16, 2026 15:20
@jsonbailey jsonbailey merged commit 39ccda7 into main Apr 16, 2026
48 checks passed
@jsonbailey jsonbailey deleted the jb/aic-2200/fix-sampled-semantics branch April 16, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants