Skip to content

Cycle: fix validation image discovery + strict ordinal parity + pytest stability #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iahncajigas merged 13 commits into from
Mar 5, 2026
Merged

Cycle: fix validation image discovery + strict ordinal parity + pytest stability #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9782abf
Simplify nSTAT-python to core package and minimal tests/docs
iahncajigas Feb 27, 2026
e0d76cd
Parity tooling: batch/resume MATLAB export and robust ordinal diffs
iahncajigas Mar 4, 2026
351a623
Cycle: fix figure-count mismatches + ordinal parity rerun + README nS…
iahncajigas Mar 5, 2026
3d97151
Cycle: fix uniqueness discovery + resolve SSIM diffs + restore pytest…
iahncajigas Mar 5, 2026
5aefe8e
chore: trigger CI for PR #2
iahncajigas Mar 5, 2026
94d2c2a
ci: sync required workflows for PR checks
iahncajigas Mar 5, 2026
4980945
ci: allow manual dispatch for test-and-build
iahncajigas Mar 5, 2026
97c2aef
ci: make required checks branch-compatible
iahncajigas Mar 5, 2026
a241343
ci: install notebook parity test deps
iahncajigas Mar 5, 2026
6661daa
ci: add nbclient dependency for validation tests
iahncajigas Mar 5, 2026
9c83239
ci: include reportlab stack for validation tests
iahncajigas Mar 5, 2026
1f1f0bc
ci: provision python3 ipykernel for notebook tests
iahncajigas Mar 5, 2026
57ee71b
Resolve conflicts with main
iahncajigas Mar 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 0 additions & 3 deletions .gitattributes
View file
Open in desktop

This file was deleted.

214 changes: 24 additions & 190 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
pull_request:
push:
branches: [main]
workflow_dispatch:

jobs:
unit-lint:
Expand All @@ -15,233 +16,66 @@ jobs:

steps:
- uses: actions/checkout@v4
with:
lfs: false

- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install package and dev dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e .[dev]

- name: Lint
run: ruff check src tests tools

- name: Type check
run: mypy src/nstat

- name: Unit tests
run: pytest

- name: Verify no MATLAB dependency
run: python tools/compliance/check_no_matlab_dependency.py
python -m pip install pyyaml nbformat nbclient reportlab pillow ipykernel
python -m ipykernel install --user --name python3 --display-name "Python 3"
- name: Run unit tests
run: pytest -q

docs-smoke-notebooks:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install package, docs, and notebook dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e .[dev,docs,notebooks]
python -m pip install reportlab pillow

- name: Resolve nSTAT data cache directory
id: nstat-data-dir
run: |
echo "dir=$(python tools/data/print_data_dir.py)" >> "$GITHUB_OUTPUT"

- name: Cache nSTAT example data
uses: actions/cache@v4
with:
path: ${{ steps.nstat-data-dir.outputs.dir }}
key: nstat-example-data-${{ runner.os }}-doi-10.6084-m9.figshare.4834640-v1
restore-keys: |
nstat-example-data-${{ runner.os }}-doi-10.6084-m9.figshare.4834640-

- name: Download nSTAT example data
run: |
python tools/data/download_example_data.py

- name: Checkout pinned MATLAB nSTAT reference
python -m pip install -e .[dev,notebooks]
python -m pip install pyyaml nbformat nbclient reportlab pillow ipykernel
python -m ipykernel install --user --name python3 --display-name "Python 3"
- name: Run notebook/visual smoke tests
run: |
python tools/parity/checkout_matlab_reference.py \
--config parity/matlab_reference.yml \
--dest /tmp/upstream-nstat \
--metadata-out parity/matlab_reference_checkout_docs_smoke.json

- name: Rebuild generated docs/notebooks
run: |
python tools/docs/generate_help_pages.py
python tools/notebooks/generate_notebooks.py
python tools/notebooks/clean_notebooks.py --check

- name: Ensure generated artifacts are committed
run: |
git diff --exit-code

- name: Build docs with warnings-as-errors
run: sphinx-build -W -b html docs docs/_build/html

- name: Verify docs search index coverage
run: python tools/docs/verify_search_index.py

- name: Run smoke notebooks
run: |
python tools/notebooks/execute_notebooks.py \
--group smoke \
--timeout 600 \
--out-report output/notebooks/notebook_execution_report.json

- name: Generate smoke validation PDF
run: |
python tools/reports/generate_validation_pdf.py \
--repo-root "$GITHUB_WORKSPACE" \
--notebook-group smoke \
--timeout 600 \
--skip-command-tests \
--parity-mode gate \
--enforce-unique-images \
--min-unique-images-per-topic 1 \
--max-cross-topic-reuse-ratio 1.0

- name: Upload smoke validation PDF artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: ci-smoke-validation-pdf
path: output/pdf/*.pdf
if-no-files-found: warn

- name: Upload smoke notebook execution artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: ci-smoke-notebook-execution
path: |
output/notebooks/notebook_execution_report.json
output/notebooks/executed/**
if-no-files-found: warn

- name: Run release gate checks
run: python tools/release/check_release_gate.py
pytest -q tests/test_helpfile_ordinal_image_parity.py tests/test_validation_images_discovery.py

matlab-data-integrity:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
lfs: false

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install integrity-check dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install pyyaml

- name: Verify mirrored MATLAB data manifest
run: |
python tools/data_mirror/verify_matlab_data.py \
--manifest data/shared/matlab_gold_20260302.manifest.json \
--strict

- name: Enforce mirror-regeneration consistency
run: |
python tools/compliance/check_mirror_regeneration.py \
--shared-root data/shared \
--allowlist tools/compliance/shared_data_allowlist.yml \
--datasets-manifest data/datasets_manifest.json
python -m pip install -e .[dev]
python -m pip install pyyaml nbformat nbclient reportlab pillow ipykernel
python -m ipykernel install --user --name python3 --display-name "Python 3"
- name: Run dataset integrity tests
run: pytest -q tests/test_datasets.py

cleanroom-compliance:
runs-on: ubuntu-latest
env:
OMP_NUM_THREADS: "1"
MKL_NUM_THREADS: "1"
OPENBLAS_NUM_THREADS: "1"
NUMEXPR_NUM_THREADS: "1"
VECLIB_MAXIMUM_THREADS: "1"

steps:
- uses: actions/checkout@v4
with:
lfs: false

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install package and compliance dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e .
python -m pip install pyyaml

- name: Checkout pinned MATLAB nSTAT reference
run: |
python tools/parity/checkout_matlab_reference.py \
--config parity/matlab_reference.yml \
--dest /tmp/upstream-nstat \
--metadata-out parity/matlab_reference_checkout.json

- name: Run clean-room overlap check
run: |
python tools/compliance/check_cleanroom_overlap.py \
--project-root "$GITHUB_WORKSPACE" \
--upstream-root /tmp/upstream-nstat \
--allowlist tools/compliance/shared_data_allowlist.yml

- name: Prepare deterministic validation images
run: |
python tools/parity/prepare_validation_images.py

- name: Build parity snapshot and enforce high/medium-severity gate
run: |
python tools/parity/build_parity_snapshot.py \
--matlab-root /tmp/upstream-nstat \
--fail-on medium

- name: Enforce Tier-1 parity progress gate
run: |
python tools/parity/check_tier1_progress.py \
--report parity/parity_gap_report.json \
--policy parity/tier1_gate_policy.yml

- name: Enforce MATLAB numeric drift thresholds
run: |
python tools/parity/build_numeric_drift_report.py \
--fixtures-manifest tests/parity/fixtures/matlab_gold/manifest.yml \
--thresholds parity/numeric_drift_thresholds.yml \
--report-out parity/numeric_drift_report.json \
--fail-on-violation

- name: Enforce functional parity progress gate
run: |
python tools/parity/check_functional_parity_progress.py \
--report parity/function_example_alignment_report.json \
--policy parity/functional_gate_policy.yml

- name: Enforce in-scope validated example statuses
run: |
python tools/parity/check_example_output_spec.py \
--report parity/function_example_alignment_report.json \
--spec parity/example_output_spec.yml

- name: Regenerate method-closure sprint backlog
run: |
python tools/parity/generate_method_closure_sprint.py \
--report parity/function_example_alignment_report.json \
--output parity/method_closure_sprint.md
git diff --exit-code parity/method_closure_sprint.md
python -m pip install -e .[dev]
python -m pip install pyyaml nbformat nbclient reportlab pillow ipykernel
python -m ipykernel install --user --name python3 --display-name "Python 3"
- name: Run API surface checks
run: pytest -q tests/test_api_surface.py
71 changes: 0 additions & 71 deletions .github/workflows/data-mirror-refresh.yml
View file
Open in desktop

This file was deleted.

28 changes: 0 additions & 28 deletions .github/workflows/docs-linkcheck.yml
View file
Open in desktop

This file was deleted.

Loading