-
Notifications
You must be signed in to change notification settings - Fork 3
Improve model_specs_are_optimal test: add supports_witnesses flag and reject large search spaces #895
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem
The `model_specs_are_optimal` test in `src/unit_tests/example_db.rs` conflates multiple failure modes into a single `else` branch. When the witness solver returns `None`, the test cannot distinguish between:
- A model that genuinely doesn't support witnesses (aggregate-only, e.g., `Sum`)
- A search space that is too large to brute-force
- A bug in the solver or registry wiring
The current fallback only checks self-consistency (`evaluate(stored_config) == stored_value`), which is too weak — it would pass even with incorrect spec data.
Goals
- No silent skips — canonical examples should always be verifiable; if they're too large for brute-force, that's a spec problem, not a test problem.
- Explicit capability detection — the test should know whether a model supports witnesses, rather than inferring it from solver failure.
- Stronger verification for aggregate-only models — even without witnesses, the test should independently verify something beyond self-consistency.
Context
Introduced by PR #805 (KthLargestMTuple) — the first aggregate-only model using `Sum`. The current workaround is functional but fragile. Concrete design decisions (e.g., where to expose `supports_witnesses`, what to verify for aggregate-only models) are open for discussion.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
Status
No status