v.1.9.1

Fixed

• Group leakage in perform_training conformal CV: now uses StratifiedGroupKFold when groups are available, matching perform_pipeline and perform_temporal_validation.

v1.9.0

Added

• FOR (False Omission Rate) metric: reported in all metrics CSVs and HTML reports
• CV+ Conformal Prediction (Mondrian):
  • Per-class prediction sets {S}, {R}, or {S, R} with distribution-free coverage guarantees
  • Computed per-fold inside nested CV
  • CV+ formal guarantee: 1 - 2α; config [Uncertainty] alpha replaces margin
  • q_hat per class saved in model bundles; HTML report with dedicated section
• Model category constants in constants.py:
  • SHAP_FALLBACK_MODELS = ("MLP", "RBF_SVC", "KNN", "TabPFN")
  • NO_CLASS_WEIGHT_MODELS = ("MLP", "KNN", "TabPFN")
• Subgroup evaluation documentation: clear guide on configuring [Metadata] section for subgroup analysis
• Conformal Prediction section in HTML report with per-model coverage tables

Changed

• Feature importance functions (has_feature_importance, get_feature_importance, compute_shap_importance) now require model_name parameter and dispatch on category constants instead of hasattr duck-typing

Removed

• Legacy config fallback for group_column in [Data] and temporal_split_column in [Validation] - use [Metadata] section (deprecated since v1.8.0)
• calculate_uncertainty() function - replaced by compute_conformal_qhat() + conformal_prediction_sets()
• [Uncertainty] margin config key - replaced by [Uncertainty] alpha
• uncertainty_margin field in model bundles - replaced by conformal_q_hat and conformal_alpha

v1.8.0

Added

• Unified [Metadata] config section with metadata column definitions:
  • group_column (moved from [Data])
  • temporal_column (moved from [Validation] as temporal_split_column)
  • subgroup_columns (new, comma-separated) for subgroup performance analysis
• Subgroup Performance Evaluation: compute full metric set (AUROC, F1, MCC, Precision, Recall, ECE, MCE, Brier Score) per subgroup value
  • Multiple subgroup columns supported simultaneously
  • Sample size and class prevalence reported per subgroup
  • Integrated with both CV and temporal validation pipelines
  • Per-subgroup metrics saved as CSV in subgroup_analysis/ directory
  • "Subgroup Analysis" section added to HTML report with tables per subgroup column
  • Warns when subgroups have fewer than 10 samples
• Signed Feature Importance Direction: determine whether features are risk factors or protective
  • New config flag: compute_feature_direction (default: false)
    • Linear models (coef_): uses sign of coefficients directly (no extra computation)
    • Tree-based models (RF, XGB, CatBoost): uses shap.TreeExplainer
    • Other models: falls back to shap.KernelExplainer for signed SHAP values
  • Direction column added to feature importance CSV (Risk (+) / Protective (-))
  • Feature importance plot color-coded by direction (firebrick = risk, seagreen = protective) with legend
  • CLI flag: respredai feature-importance --directionv1.8.0

v1.7.1

Added

• BCa Bootstrap Confidence Intervals: replaced percentile bootstrap with bias-corrected and accelerated (BCa) method via scipy.stats.bootstrap for improved coverage on small samples and bounded/skewed metrics
• Nadeau-Bengio Corrected Standard Error: new SE column in metrics CSV using the corrected variance formula (1/k + n_test/n_train) * s**2 that accounts for training set overlap in k-fold CV. Summary report ± notation now uses SE instead of raw Std
• Configurable bootstrap CI parameters: confidence_level and n_bootstrap in [Pipeline] config section
• Constants module (respredai/core/constants.py): centralized validation lists, directory names, and defaults

Fixed

• Name sanitization unified across 8 files - .replace(" ", "_") and re.sub() patterns consolidated into sanitize_name() / sanitize_metric_name() (44 replacements)
• assert in temporal split replaced with proper ValueError
• Empty CV fold validation with warning when train/test sets are empty after splitting
• Explicit warning when all bootstrap samples fail (previously returned NaN silently)

Changed

• ConfigHandler split into domain-specific dataclasses for clarity
• Config validation lists now reference centralized constants from constants.py
• README quick-start config example now shows [Validation] section (added in v1.7.0)
• HTML report confidence intervals row now dynamically reflects configured confidence_level and n_bootstrap

v1.7.0

Added

• Temporal (Prospective-Style) Validation:
  • validation_strategy config option: cv (default), temporal, or both
  • temporal_split_column, temporal_split_date, and temporal_split_ratio config options
  • Group-aware temporal splitting to prevent data leakage
  • --validation-strategy CLI override flag for the run command
  • Temporal validation results section in HTML report
• Per-fold One-Hot Encoding: OHE is now fitted inside each CV fold (train-only) instead of on the full dataset, preventing category leakage
• NaN-safe scaling for KNN imputation: pre-scales features with NaN-tolerant statistics before distance-based imputation
• scale_pos_weight parameter in XGBoost hyperparameter grid

Fixed

• SVC models now set probability=False when external calibration is enabled, avoiding double Platt scaling
• TunedThresholdClassifierCV (CV method) now uses StratifiedGroupKFold and passes group labels when grouped CV is configured
• Repeated CV deduplication now uses the threshold-aware decision boundary instead of a fixed 0.5 cutoff
• All nanstd calls now use ddof=1 for unbiased sample standard deviation
• Reliability curve binning replaced with self-contained implementation to ensure bin_counts alignment
• Logger initialization deferred to after CLI overrides so log file uses the correct output folder
• Feature importance name resolution improved with multi-source fallback; missing features across folds default to zero
• evaluate command now reuses the saved OHE transformer from training instead of ad-hoc pd.get_dummies

Changed

• validate-config summary table now displays validation strategy and temporal split parameters
• create-config template now includes a commented [Validation] section
• uncertainty_margin now stored in training metadata and model bundles

v1.6.2

Fixed

• Bootstrap confidence intervals now deduplicate samples when using repeated outer CV
• Threshold optimization (CV method) now correctly uses the calibrated estimator when probability calibration is enabled
• Metrics aggregation now respects repeat structure
• Reliability curve fold labels now indicate repeat number when using repeated CV
• Reliability curves now use quantile binning for smoother calibration plots on imbalanced data

Added

• Makefile for development workflows

v1.6.1

Fixed

• train subcommand now applies probability calibration (CalibratedClassifierCV) when calibrate_probabilities = true
• train subcommand now supports CV threshold method (TunedThresholdClassifierCV) in addition to OOF
• Reproducibility manifest now includes probability calibration parameters

Changed

• create-config template now includes threshold_objective, vme_cost, me_cost parameters
• validate-config summary table now displays probability calibration and threshold objective details

v1.6.0

Added

• Probability Calibration:

  • Optional post-hoc probability calibration on the best estimator per outer CV fold
  • Supports sigmoid (Platt scaling) and isotonic calibration methods
  • Applied after hyper-parameters tuning and before threshold tuning

• Calibration Diagnostics:

  • Brier Score: Mean squared error of probability predictions (lower is better)
  • ECE (Expected Calibration Error): Weighted average of calibration error across bins
  • MCE (Maximum Calibration Error): Maximum calibration error across any bin
  • Reliability curves (calibration plots) per outer CV fold and aggregate

• Repeated Stratified Cross-Validation:

  • outer_cv_repeats config option (default: 1)
  • Set >1 for repeated CV with different shuffles for more robust performance estimates

Changed

• metric_dict() now includes Brier Score, ECE, and MCE by default
• HTML report includes new calibration diagnostics section
• Output folder now includes calibration/ directory with reliability curve images

v1.5.1

Added

• OneHotEncoder min_frequency parameter to reduce noise from rare categorical values

Changed

• Updated requirements.txt with explicit version constraints for all dependencies
  • scikit-learn>=1.5.0 required for TunedThresholdClassifierCV

v1.5.0

Added

• VME/ME report:
  • VME (Very Major Error): Predicted susceptible when actually resistant
  • ME (Major Error): Predicted resistant when actually susceptible
• Flexible threshold objectives:
  • threshold_objective config option: youden (default), f1, f2, cost_sensitive
  • Cost-sensitive optimization with configurable vme_cost and me_cost weights
• Per-prediction uncertainty quantification to flag uncertain predictions near decision threshold
  • uncertainty_margin config option (default: 0.1) defines margin around threshold
  • Predictions within margin are flagged as uncertain in evaluation output
  • Uncertainty scores (0-1) provided for each prediction
• Reproducibility manifest (reproducibility.json) generated with run and train commands with environment info, data fingerprint, full configuration settings

Changed

• HTML report framework summary now displays threshold objective and cost weights (when applicable)
• Evaluation output now includes uncertainty and is_uncertain columns