feat: cross-language conformance test suite (#155)

CHANGELOG.md

@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Cross-language conformance test suite** ([#155](https://github.com/vig-os/fd5/issues/155))
+  - 6 canonical fixture generators: minimal, sealed, with-provenance, multiscale, tabular, complex-metadata
+  - 3 invalid fixture generators: missing-id, bad-hash, no-schema
+  - Expected-result JSON files defining the format contract for any language binding
+  - 39 pytest conformance tests covering structure, hash verification, provenance, multiscale, tabular, metadata, schema validation, and negative tests
+  - README documenting how to use the suite and add new cases
+
 - **Preflight feedback and status dashboard for devc-remote** ([#149](https://github.com/vig-os/fd5/issues/149))
   - Each preflight check now prints a success/warning/error status line as it completes
   - New checks: container-already-running, runtime version, compose version, SSH agent forwarding

tests/conformance/README.md

@@ -0,0 +1,70 @@
+# Cross-Language Conformance Test Suite
+
+Language-agnostic test suite for the fd5 format. Any fd5 implementation
+(Python, Rust, Julia, C/C++, TypeScript) must pass these tests to prove
+format conformance.
+
+## Structure
+
+```
+tests/conformance/
+├── README.md                  # This file
+├── generate_fixtures.py       # Regenerates .fd5 fixture files
+├── test_conformance.py        # Python conformance runner
+├── fixtures/                  # Generated .fd5 files (not checked in)
+├── expected/                  # Expected-result JSON (checked in)
+│   ├── minimal.json
+│   ├── with-provenance.json
+│   ├── multiscale.json
+│   ├── tabular.json
+│   ├── complex-metadata.json
+│   └── sealed.json
+└── invalid/                   # Invalid .fd5 files + expected errors
+    └── expected-errors.json
+```
+
+## How It Works
+
+1. `generate_fixtures.py` uses the Python reference implementation to create
+   canonical `.fd5` fixture files in `fixtures/` and invalid files in `invalid/`.
+2. Each fixture has a corresponding JSON file in `expected/` that defines the
+   expected root attributes, dataset shapes, dtypes, group hierarchy, etc.
+3. A conformance runner opens each fixture with the language's own reader,
+   extracts values, and asserts equality against the expected JSON.
+
+## Running (Python)
+
+```bash
+uv run pytest tests/conformance/ -v
+```
+
+Fixtures are auto-generated by a pytest session-scoped fixture before tests run.
+
+## Adding a New Conformance Case
+
+1. Add a generator function in `generate_fixtures.py`.
+2. Create a corresponding `expected/<name>.json` with the expected structure.
+3. Add test functions in `test_conformance.py` (or the equivalent in your language).
+4. Run the suite to verify.
+
+## Test Categories
+
+| Category              | What it tests                                          |
+|-----------------------|--------------------------------------------------------|
+| Structure             | Correct group hierarchy, required attributes present   |
+| Data round-trip       | Write values, read back, compare dtype/shape/values    |
+| Hash verification     | Sealed files verify; tampered files fail               |
+| Provenance            | DAG traversal returns expected source chain            |
+| Schema validation     | Embedded schema validates the file's own structure     |
+| Negative tests        | Invalid files are rejected with appropriate errors     |
+
+## For Other Languages
+
+To implement the conformance suite in a new language:
+
+1. Generate fixtures using the Python script (or use pre-generated ones from CI).
+2. Load each `.fd5` file with your HDF5 library.
+3. Parse the corresponding `expected/*.json`.
+4. Assert that the extracted values match the expected JSON.
+
+This is a black-box test -- it tests the format contract, not internal APIs.

tests/conformance/expected/complex-metadata.json

@@ -0,0 +1,46 @@
+{
+  "description": "Deeply nested metadata groups — metadata tree tests",
+  "root_attrs": {
+    "product": "test/conformance",
+    "name": "complex-metadata-conformance",
+    "description": "Complex metadata conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [4, 4],
+      "dtype": "float32"
+    }
+  ],
+  "groups": [
+    "/",
+    "/metadata",
+    "/metadata/acquisition",
+    "/metadata/reconstruction",
+    "/metadata/reconstruction/parameters"
+  ],
+  "verify": true,
+  "metadata_tree": {
+    "metadata": {
+      "version": 2,
+      "acquisition": {
+        "modality": "PET",
+        "duration_sec": 300.0,
+        "isotope": "F-18"
+      },
+      "reconstruction": {
+        "algorithm": "osem",
+        "parameters": {
+          "iterations": 4,
+          "subsets": 21
+        }
+      }
+    }
+  }
+}

tests/conformance/expected/minimal.json

@@ -0,0 +1,26 @@
+{
+  "description": "Smallest valid fd5 file — structure tests",
+  "root_attrs": {
+    "product": "test/conformance",
+    "name": "minimal-conformance",
+    "description": "Minimal conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [4, 4],
+      "dtype": "float32"
+    }
+  ],
+  "groups": [
+    "/"
+  ],
+  "verify": true,
+  "schema_valid": true
+}

tests/conformance/expected/multiscale.json

@@ -0,0 +1,45 @@
+{
+  "description": "File with pyramid/multiscale datasets — multiscale tests",
+  "root_attrs": {
+    "product": "recon",
+    "name": "multiscale-conformance",
+    "description": "Multiscale conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "groups": [
+    "/",
+    "/pyramid",
+    "/pyramid/level_1",
+    "/pyramid/level_2"
+  ],
+  "pyramid": {
+    "n_levels": 2,
+    "scale_factors": [2, 4],
+    "level_shapes": {
+      "level_1": [4, 4, 4],
+      "level_2": [2, 2, 2]
+    }
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [8, 8, 8],
+      "dtype": "float32"
+    },
+    {
+      "path": "/mip_coronal",
+      "dtype": "float32"
+    },
+    {
+      "path": "/mip_sagittal",
+      "dtype": "float32"
+    }
+  ],
+  "verify": true,
+  "schema_valid": true
+}

tests/conformance/expected/sealed.json

@@ -0,0 +1,28 @@
+{
+  "description": "File with verified content hash — hash verification tests",
+  "root_attrs": {
+    "product": "test/conformance",
+    "name": "sealed-conformance",
+    "description": "Sealed conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [8, 8],
+      "dtype": "float32"
+    }
+  ],
+  "verify": true,
+  "schema_valid": true,
+  "hash_verification": {
+    "intact_verifies": true,
+    "tampered_attr_fails": true,
+    "tampered_data_fails": true
+  }
+}

tests/conformance/expected/tabular.json

@@ -0,0 +1,36 @@
+{
+  "description": "Compound dataset (event table) — tabular data tests",
+  "root_attrs": {
+    "product": "test/conformance",
+    "name": "tabular-conformance",
+    "description": "Tabular conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [4, 4],
+      "dtype": "float32"
+    },
+    {
+      "path": "/events",
+      "shape": [5],
+      "columns": ["time", "energy", "detector_id"]
+    }
+  ],
+  "verify": true,
+  "tabular": {
+    "row_count": 5,
+    "column_names": ["time", "energy", "detector_id"],
+    "column_dtypes": {
+      "time": "float64",
+      "energy": "float32",
+      "detector_id": "int32"
+    }
+  }
+}

tests/conformance/expected/with-provenance.json

@@ -0,0 +1,46 @@
+{
+  "description": "File with source links — provenance tests",
+  "root_attrs": {
+    "product": "test/conformance",
+    "name": "provenance-conformance",
+    "description": "Provenance conformance fixture",
+    "timestamp": "2026-01-01T00:00:00Z",
+    "_schema_version": 1
+  },
+  "root_attrs_prefixed": {
+    "id": "sha256:",
+    "content_hash": "sha256:"
+  },
+  "datasets": [
+    {
+      "path": "/volume",
+      "shape": [4, 4],
+      "dtype": "float32"
+    }
+  ],
+  "groups": [
+    "/",
+    "/sources",
+    "/sources/upstream",
+    "/provenance",
+    "/provenance/ingest"
+  ],
+  "verify": false,
+  "provenance": {
+    "sources": [
+      {
+        "name": "upstream",
+        "id": "sha256:aaa111",
+        "product": "raw",
+        "role": "input_data",
+        "description": "Upstream raw data"
+      }
+    ],
+    "has_original_files": true,
+    "original_files_count": 1,
+    "ingest": {
+      "tool": "conformance_generator",
+      "tool_version": "1.0.0"
+    }
+  }
+}

tests/conformance/fixtures/.gitignore

@@ -0,0 +1,2 @@
+*.fd5
+*.h5