-
Notifications
You must be signed in to change notification settings - Fork 0
[P0] Define eval runtime provenance and requested-vs-actual semantics #30
Copy link
Copy link
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or request
Description
Objective
Specify how eval tooling records requested versus actual runtime and model metadata.
Priority
P0 — Must Fix for v1.0
Details
The eval report format already distinguishes config values from observed runtime metadata, but the spec does not define the source-of-truth order between API responses, runtime JSON output, CLI metadata, and requested config/env values. This makes report provenance underspecified.
Acceptance Criteria
- The spec defines the precedence order for runtime/model provenance
- Requested and observed fields are defined consistently in eval report prose
- Fallback behavior is defined for runtimes without model introspection support
Notes
Source: work/eval-design-discussion.md
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or request