Skip to content

Improve model_specs_are_optimal test: add supports_witnesses flag and reject large search spaces #895

@isPANN

Description

@isPANN

Problem

The `model_specs_are_optimal` test in `src/unit_tests/example_db.rs` conflates multiple failure modes into a single `else` branch. When the witness solver returns `None`, the test cannot distinguish between:

  • A model that genuinely doesn't support witnesses (aggregate-only, e.g., `Sum`)
  • A search space that is too large to brute-force
  • A bug in the solver or registry wiring

The current fallback only checks self-consistency (`evaluate(stored_config) == stored_value`), which is too weak — it would pass even with incorrect spec data.

Goals

  1. No silent skips — canonical examples should always be verifiable; if they're too large for brute-force, that's a spec problem, not a test problem.
  2. Explicit capability detection — the test should know whether a model supports witnesses, rather than inferring it from solver failure.
  3. Stronger verification for aggregate-only models — even without witnesses, the test should independently verify something beyond self-consistency.

Context

Introduced by PR #805 (KthLargestMTuple) — the first aggregate-only model using `Sum`. The current workaround is functional but fragile. Concrete design decisions (e.g., where to expose `supports_witnesses`, what to verify for aggregate-only models) are open for discussion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions