build-cd v8: Infrastructure Update for spack v1.X

[CodeGat]

Closes #313
References ACCESS-NRI/access-spack-packages#295, ACCESS-NRI/schema#53

Important

This PR is a major update to the deployment infrastructure. See below for the prerequisites for this repository to be able to merge this PR.

Important

This major version change marks the end of major infrastructure updates for deployments to spack < 1.0.
If you want to deploy to instances of spack < 1.0, you must use build-cd <= v7.
If you want to deploy to instances of spack >= 1.0, you must use build-cd >= v8.

Background

Finally, the migration to spack v1.X is here!

This requires a bunch of changes to the deployment infrastructure, mostly because spack has decoupled their builtin spack/spack-packages repository from the history of the spack repository. This means that for provenance we now need to keep track of the new spack/spack-packages (the builtin spack packages repository) and an updated ACCESS-NRI/access-spack-packages (renamed from ACCESS-NRI/spack-packages).

This change has ramifications across a bunch of repos: all MDRs, ACCESS-NRI/schema, ACCESS-NRI/[access-]spack-packages, and ACCESS-NRI/spack-config. It also requires changes to other repositories, see the linked project.

Importantly (as noted in the !IMPORTANT note!), this major version change is a cut-off for deployments to spack < 1.0. This means that if you want to deploy to spack 0.22 instances, for example, you must stay on build-cd <= v7 in your MDRs. This was done to simplify the infrastructure, so there isn't a bunch of "run this step if spack < v1, etc".

The PR

This PR has a few main areas of change.

Areas of Change hidden in this section - there is a lot!

Updated the CI/CD Deployment Logic

• Remove the bespoke spack-packages updating logic in the deployment step, in favour of a spack-managed spack repo update command, done in b5f5db1
• In the setup deployment phase: validate the SHAs for all repository references in both build-cd config/settings.json and the MDRs config/versions.json using access-nri/actions get-git-ref-info action (from Add get-git-ref-info action, update READMEs actions#39). Done in 3e7bb4c
• Updated the deployment PR comment and release notes in 8257842, and the associated tests in e63e800
• Prerelease injection: Instead of the spack-specific repos:: key, use a normal repos: key. Updated logic in ae27874

Updated the CD Model Provenance Upload

• Added the extra fields for access and builtin spack packages
• Updated the tracking_services schema version to 2-0-0 - will need to update the schema as in Schema updates for spack v1 schema#57
• Reorganised and updated tests

Done in 65e20bb

Updated the Create Deployment Spack Workflow

• When creating a new instance of spack using this workflow, our spack-packages repository is now cloned when spack bootstraps (see ACCESS-NRI/spack-packages as git-based packages repo spack-config#82). Since this is the case, we no longer need to reference it. See 0da9567

Updated the Cleanup Logic

• We remove the spack-packages repo removal step in the post-PR cleanup job. This is because we've moved the Prerelease-specific access-spack-packages repository clone destination into the environment itself, which means the spack env rm command removes the access-spack-packages repo also. Done in f82ca40

Updated the config/settings.json Deployment Logic

• Updated the config/settings.json logic to be able to update the builtin spack-packages repository as well, as we now keep track (and modify) that repository from build-cd centrally.
• Also updated the config/settings.schema.json to add the new deployment.TARGET.TYPE.VERSION.builtin-spack-packages field for VERSION >= 1, data now looks like:

{
  "$schema": "./settings.schema.json",
  "deployment": {
    "Gadi":
      # ...
      "Prerelease": {
        "0.22": {
          "spack": "357ff5dde01a32612911fd2021e1feefd33a3a3c",
          "spack-config": "2025.08.000"
-       }
+       },
+       "1.0": {
+         "spack": "357ff5dde01a32612911fd2021e1feefd33a3a3c",
+         "builtin-spack-packages": "2025.07.0",
+         "spack-config": "2025.08.000"
+       }
      }
    }
  }
}

Relevant changes in https://github.com/ACCESS-NRI/build-cd/pull/326/files/0da9567a5f5021c5cbe3bab1253ac1539d97b95a..0aa2c7fa2cc377ce66e2019fedafac49f67ca7a7

Misc Changes...

• The validate-repo-version action was deleted as it is now unreferenced, replaced by ACCESS-NRI/actions get-git-ref-info action in Add get-git-ref-info action, update READMEs actions#39, see 6622773

• Intra-workflow refs were updated to v8 in 17aa304

Other PRs Merged Into This One

• Spack V1 Migration Feature: Reserved Spack Definitions #332
• Spack V1 Migration Feature: config/versions.json custom-scopes Field #335
• Remove Entrypoint inputs.root-sbd, Add inputs.spack-manifest-path Where Missing #347

Testing

MDRs (TC 1) ✔️

MDR Testing details hidden in this section

Tested in ACCESS-NRI/ACCESS-TEST#63

Prerelease (Commit-Based) (TC 1.1) ✔️

Success, see commit and run.

Prerelease (!redeploy-Based) (TC 1.2) ✔️

Success, see comment and run.

Prerelease (Cleanup) (TC 1.3) ✔️

Success, see event and run.

Prerelease (Multi-Spec) (TC 1.4) ✔️

Infrastructure side of things was a success - concretization failed as we don't have anything to hinge multi-spec builds on in ACCESS-TEST but the injections were valid. See commit and correct injection.

Prerelease (Custom Scopes) (TC 1.5) ✔️

Success, see commit and run.

Release (TC 1.6) ✔️

Success, although metadata upload failed as the database is not equipped to read it yet. See commit and run, and the release.

SDRs (TC 2) ✔️

SDR Testing details hidden in this section

Tested in ACCESS-NRI/system-tools#21

Prerelease (Commit-Based) (TC 2.1) ✔️

Success, see commit and run.

Prerelease (!redeploy-Based) (TC 2.2) ✔️

Success, see comment and run.

Prerelease (Cleanup) (TC 2.3) ✔️

Success, see event and run.

Prerelease (Custom Modulefiles) (TC 2.4) ✔️

Success, see commit and run.

Prerequisites For Merging

• Requires build-cd v7: !update-configs: Prerelease to Model Configs Pipeline #321 to be merged, and both this branch and v8 rebased onto v7
• Requires ACCESS-NRI/spack-packages as git-based packages repo spack-config#82 to be merged
• Requires Schema updates for spack v1 schema#57 to be merged
• Requires Consider renaming repository "spack-packages" to "access-spack-packages" access-spack-packages#295 to be closed
• Requires all inter-workflow refs to be updated to v8
• Update all MDRs/SDRs/templates entrypoint inputs where required

[CodeGat]

That's all the PRs! Now onto E2E Testing...

[CodeGat]

Just so we don't have to scroll up...

Testing

MDRs (TC 1) ✔️

MDR Testing details hidden in this section

Tested in ACCESS-NRI/ACCESS-TEST#63

Prerelease (Commit-Based) (TC 1.1) ✔️

Success, see commit and run.

Prerelease (!redeploy-Based) (TC 1.2) ✔️

Success, see comment and run.

Prerelease (Cleanup) (TC 1.3) ✔️

Success, see event and run.

Prerelease (Multi-Spec) (TC 1.4) ✔️

Infrastructure side of things was a success - concretization failed as we don't have anything to hinge multi-spec builds on in ACCESS-TEST but the injections were valid. See commit and correct injection.

Prerelease (Custom Scopes) (TC 1.5) ✔️

Success, see commit and run.

Release (TC 1.6) ✔️

Success, although metadata upload failed as the database is not equipped to read it yet. See commit and run, and the release.

SDRs (TC 2) ✔️

SDR Testing details hidden in this section

Tested in ACCESS-NRI/system-tools#21

Prerelease (Commit-Based) (TC 2.1) ✔️

Success, see commit and run.

Prerelease (!redeploy-Based) (TC 2.2) ✔️

Success, see comment and run.

Prerelease (Cleanup) (TC 2.3) ✔️

Success, see event and run.

Prerelease (Custom Modulefiles) (TC 2.4) ✔️

Success, see commit and run.

[aidanheerdegen]

Mammoth effort! Thank you so much for wrangling all this.