Skip to content

Comments

Feat/standardize benchmarks#29

Open
miagarvey wants to merge 7 commits intomainfrom
feat/standardize-benchmarks
Open

Feat/standardize benchmarks#29
miagarvey wants to merge 7 commits intomainfrom
feat/standardize-benchmarks

Conversation

@miagarvey
Copy link
Collaborator

Standardize fa, pheno, drug, study params benchmarks

⚙️ Release Notes

  • Create shared_utils.py with shared evaluation functions (exact_match, semantic_similarity, category_equal, compute_weighted_score); refactor FA, Drug, and Study Parameters benchmarks to use them.
  • Add optional field_weights, dependency validation (Drug: direction/association checks; Study Parameters: statistical consistency), and standardized allele handling (semantic_similarity) across all benchmarks.
  • Standardize Pheno benchmark: return detailed dict (0-1 scale) instead of single float (0-100), use field-specific evaluators from shared_utils, add field_weights, and update scaffold to accept ground truth directly (backward compatible with PMCID fallback).

By submitting this pull request, you agree to follow our Coding Guidelines:

- Create shared_utils.py with common evaluation functions
- Add configurable field_weights parameter to all benchmarks
- Add dependency validation to Drug and Study Parameters benchmarks
- Standardize allele handling to use semantic similarity
- Add statistical consistency validation to Study Parameters
- Refactor all benchmarks to use shared utilities
- Maintain backward compatibility with existing code
- Create shared_utils.py with common evaluation functions
- Add configurable field_weights parameter to all benchmarks
- Add dependency validation to Drug and Study Parameters benchmarks
- Standardize allele handling to use semantic similarity
- Add statistical consistency validation to Study Parameters
- Refactor all benchmarks to use shared utilities
- Maintain backward compatibility with existing code
…led results dict, 0-1 scale, shared_utils evaluators, configurable field_weights, direct ground truth parameter
@miagarvey miagarvey closed this Nov 13, 2025
@miagarvey miagarvey reopened this Nov 13, 2025
…alysis summary of first 5 articles, fully removed variant id and study params id from study params score, modified pheno alignment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant