test: tighten mock contract + expand e2e.yml so future #207-class bugs cannot ship

[gerchowl]

Context

#207 (split into PRs #208 and #209) was a real contract bug — per-file bake/derive didn't emit `release-manifest.json` — that all four client unit suites and the Python E2E missed. Caught only on a live anvil-dev dagger run hitting the JS/shell/Rust clients. The user asked the right question: can mocks catch this in the future, and is `mat-vis-tst` the next-level test that should have caught it?

Both, yes. This issue does both.

Part A — sharper unit-test contract

Mocks DID exercise `bake_one_per_file` and `derive_smaller_tier` end-to-end, but only asserted the happy path (textures committed, sentinel last). They didn't assert WHICH manifest entries are produced, so a missing `release-manifest.json` op was invisible.

Adds

• `tests/test_hf_bake_per_file.py::test_commit_operations_include_release_manifest`:

  • Run `bake_one_per_file` against mocked HfApi
  • Assert: across `api.create_commit.call_args_list`, exactly one op writes `path_in_repo == "release-manifest.json"`
  • Assert: that commit's ops also contain the source catalog (`.json`)
  • Assert: the manifest commit carries `parent_commit` (CAS retry primitive)

• `tests/test_hf_derive_per_file.py::test_commit_operations_include_release_manifest_after_derive`:

  • Same shape; covers the derive path's manifest emission

• A small substrate-contract module `tests/_substrate_contract.py` (or a fixture) that lists the required commit-path-set for any bake/derive completion: `{.json, release-manifest.json, //.tier_complete}`. Both bake and derive tests assert their commit history covers this set. Future contract changes update the fixture once.

Part B — `mat-vis-tst` as the gate

`mat-vis-tst` exists for exactly this — catching contract slips that mocks can't. `tests/e2e/test_per_file_roundtrip.py` already exists (#193) but currently only:

• Bakes ONE source (`polyhaven`)
• Tests the Python client only
• Runs nightly via `e2e.yml` from test(e2e): mat-vis-tst smoke tests — multi-language round-trip per ADR-0012 #205 (newly added)

Expands

`tests/e2e/test_per_file_roundtrip.py` matrix:

• Bake two sources (polyhaven + ambientcg) sequentially against one throwaway tag → exercises `_fetch_manifest_with_parent` merge
• Derive one source (polyhaven 1k → 512) → exercises derive-side manifest emission (fix(baker): per-file bake must write release-manifest.json (live JS/shell/Rust 404 on E2E) #207 part 2)
• Assert manifest accumulates correctly: `sources.polyhaven.tiers == {1k, 512}` and `sources.ambientcg.tiers == {1k}`

`e2e.yml` workflow extension:

• After the existing Python E2E job, fan out one job per language (JS, shell, Rust) with the throwaway tag in the env. Each fetches a real per-file PNG/byte-stream and asserts magic.
• Final job: `mat-vis-baker audit-orphans --repo gerchowl/mat-vis-tst` and assert `orphans == 0` post-cleanup

Why this matters

The Python client masks manifest 404s with tree-fallback (`client.py:544`); the JS/shell/Rust clients hard-fail. Without exercising those clients against real bytes from a fresh bake, contract slips on the manifest are invisible until ship-day.

Acceptance

• Unit tests assert manifest path appears in commit ops for both bake and derive
• Unit tests assert `parent_commit` is set on the manifest commit (CAS retry)
• E2E runs the multi-source bake + derive scenario locally with `MAT_VIS_E2E=1 pytest tests/e2e/`
• e2e.yml fans out JS / shell / Rust jobs after the Python suite
• e2e.yml includes a final `audit-orphans` assertion
• Live anvil-dev re-run confirms green nightly flow

Estimate

~80 LOC across 3 test files + ~40 LOC YAML for the workflow. Small, mostly mechanical; benefits compound across the whole milestone.

[gerchowl]

Concurrency / CAS-retry gap (raised in chat)

Real gap I missed in the original Part A — neither the 412-retry loop nor the parallel-write race is tested anywhere. Only `_merge_manifest_for_source` (the pure merge function) has unit coverage.

Add to Part A

`tests/test_hf_bake_per_file.py`:

• `test_manifest_commit_retries_on_412` — patch `HfApi.create_commit` to raise once with a "412 Precondition Failed" message, then succeed. Assert: retry counter incremented, `_fetch_manifest_with_parent` called twice, final manifest body contains the source.
• `test_manifest_commit_exhausts_retries` — patch to always raise 412. Assert: raises after `max_retries` attempts, parent_commit kwarg present on each attempt.
• `test_manifest_commit_does_not_retry_on_non_412` — patch to raise a 401 / generic exception. Assert: re-raises immediately, only one attempt.

`tests/test_hf_derive_per_file.py`:

• Mirror the three above for the derive path's catalog+manifest commit.

Add to Part B (truly parallel)

`tests/e2e/test_per_file_concurrent_writes.py` — opt-in via `MAT_VIS_E2E=1`:

• Use `multiprocessing.Process` (NOT threading — Python's GIL serializes the live HTTP and never fires the retry path; verified during the fix(baker): per-file bake must write release-manifest.json (live JS/shell/Rust 404 on E2E) #207 fix). Two processes call `bake_one_per_file` against the same release tag with different sources at the same instant.
• Assert: both bakes' `result["ok"] > 0`, final manifest contains BOTH sources, `audit-orphans` reports 0 orphans.
• Auto-clean throwaway tag.

Add to Part B's `e2e.yml`

The matrix bake step in `bake.yml` is already a multi-job CI matrix. `e2e.yml` should mirror that — kick off `dagger call bake` jobs for polyhaven and ambientcg in parallel against one tag, then assert the final manifest has both sources. This is the closest we can get to "real prod matrix" in CI; it'll catch any regressions in `parent_commit` handling against actual HF servers.

Why this matters

The original CAS implementation in #208 was untested for the retry path — a regression that silently dropped the `parent_commit` parameter would have shipped through the unit tests. The mocked happy path passed without ever exercising the lock semantics.

[gerchowl]

Design decision: stay with CAS retry (A); file fragments+merge (B) as escape hatch only if needed

User raised the alternative: instead of CAS retry, have each matrix worker write its own manifest fragment (manifest-fragment-<source>-<tier>.json), then a final needs: matrix job merges all fragments into release-manifest.json.

Why A wins for v0.6.0

• Worst-case load: ≤4 textured sources × ≤4 tiers = ~16 concurrent writers. CAS with 6 retries comfortably clears that.
• HF Hub's entire commit API is built around parent_commit semantics; it's the documented hot path.
• B introduces a coordinator job — extra YAML, single-point-of-failure (one fragment lost → no manifest at all).
• Both A and B keep clients reading one static file; A doesn't touch the client contract at all.

When B becomes the right answer

When empirical CAS retry rate exceeds ~5%, or we ever see exhaustion in prod. We won't know that without measurement.

Concrete addition to this issue

Adding a fourth test deliverable: tests/e2e/test_per_file_concurrent_writes.py (opt-in via MAT_VIS_E2E=1 MAT_VIS_E2E_CONCURRENCY=8):

• Fire N (default 8) multiprocessing.Process workers; each calls bake_one_per_file against the same release tag with a different source/tier slug.
• Capture retry counts via a structured log (we'll add a manifest_cas_retry event in fix(baker): emit release-manifest.json with CAS retry (#207) #208's fix).
• Assert: all workers succeed, final manifest has every source/tier, retry count stays under max_retries / 2 per worker on average (alarm at half-budget = signal to investigate).
• If the assertion fires repeatedly in CI, that's the trigger to file the B follow-up issue with measurement attached.

So: measurement before redesign. If A holds, ship it. If it doesn't, B has a paved path.
EOF
)

[gerchowl]

Workflow matrix expansion (fan-out JS/shell/Rust to e2e.yml) deferred from PR #211 — needs a workflow refactor since the Python E2E fixture auto-cleans its tag. Filing as a small follow-up; pure YAML/dagger plumbing, no new test logic.