Open
Conversation
…sues, remove CatBoost refs
Add MetaAttack(Attack) with validated constructor, _parse_attacks(), and abstract method stubs. Register as "meta" in the attack factory. Supports (name, params, n_reps) tuples with validation against supported attacks (lira, qmia, structural). Loads k-anonymity threshold from ACRO config when not explicitly provided. Includes design spec and staged implementation plan.
Add _run_sub_attack() and the orchestration loop in _attack(). Each sub-attack runs in an isolated subdirectory under output_dir to prevent shadow model and report collisions between runs. MIA attacks (LiRA, QMIA) get report_individual=True injected automatically. Structural always computes record-level results. Sub-attack objects are collected for score extraction in Stage 3.
Add _extract_mia_scores() and _extract_structural_scores() with a field-mapping dict (_MIA_SCORE_FIELDS) for LiRA/QMIA score paths. Wire extraction into _attack() loop: scores collected immediately after each sub-attack run into mia_scores and structural_scores dicts, keyed by attack name with one list per repetition.
- Guard against sub-attack not running: check return value from attack() and raise RuntimeError with clear message if empty - Reject empty attacks list in _parse_attacks with ValueError - Use copy.deepcopy(params) instead of shallow dict(params) to prevent nested mutable values leaking between repetitions - Add logging.basicConfig to match peer attack file conventions
Implement _build_dataframe() with: - Level 1 (within-attack): mean, std, consistency per MIA attack across n_reps; mean k / majority vote for structural reps - Level 2 (cross-attack): arithmetic and geometric mean of MIA per-attack means; binary structural flag; n_vulnerable count - NaN padding for structural columns on test records - Epsilon-stabilised geometric mean to handle log(0) Wire into _attack(): DataFrame stored on self.vulnerability_df after all sub-attacks complete and scores are extracted.
- Clip MIA scores to [0, 1] during extraction to handle LiRA Carlini modes that produce unbounded log-likelihood ratios - Document LiRA score convention: score = CDF under out-distribution, high values = evidence for membership (not against) - Replace 'v is True' identity check with truthiness test 'if v:' to handle numpy bools correctly - Round averaged k-anonymity to int for multi-rep structural runs (fractional k is not meaningful)
Complete the MetaAttack pipeline: - _compute_global_metrics: uses mia_mean as membership predictor with get_metrics() for AUC/TPR/Advantage; falls back to summary dict for structural-only configs - _construct_metadata: enriches report with thresholds and key metrics - _get_attack_metrics_instances: standard report structure with sub-attack summary and full DataFrame under "individual" - CSV export: saves vulnerability_matrix.csv alongside JSON report - _attack() now returns a proper report dict (no more NotImplementedError)
10 test cases covering: - Validation: unsupported attack, invalid tuple, empty list, bad n_reps - Integration: QMIA + structural basic run, DataFrame shape and columns - Structural NaN for test records - Repeated runs: std column exists, consistency in [0, 1] - Threshold effects: lower threshold flags more records - Global metrics: AUC and TPR in [0, 1] - Report structure: standard nested JSON keys - Factory integration: factory.attack(target, "meta", ...) works - CSV export: vulnerability_matrix.csv written and loadable
Demonstrates end-to-end usage: synthetic data, Target construction, MetaAttack with QMIA (2 reps) + structural, DataFrame inspection, summary statistics, and top-10 most vulnerable records.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #441 +/- ##
==========================================
- Coverage 99.51% 99.10% -0.42%
==========================================
Files 23 25 +2
Lines 2692 3009 +317
==========================================
+ Hits 2679 2982 +303
- Misses 13 27 +14 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #428
Added a MetaAttack class that runs multiple privacy attacks on the same target and combines their per-record results into one DataFrame.
vulnerability_matrix.csvand a standard JSON reportavailable via the factory:
factory.attack(target, "meta", attacks=[...])