`git.BRANCH` References Are Not Picking Up Changes In Linked Repository

[CodeGat]

Similar to #2.

Background

Code changes to the underlying branch in a git.BRANCH reference aren't being picked up during a new install of a model. Even with a workaround that updated the underlying repositories the install still reused an existing package.

When building this in a separate instance manually, the code changes are picked up, and the model fails to build (which is expected, as there was an error in the code).

Workarounds

Currently, the only workaround seems to be using a git.TAG or git.COMMIT reference, rather than a git.BRANCH reference.

Implementation Plan

Copy the instance that is having an issue, and then:

• Investigate the spack solve of the access-esm1p6 package to figure out why it is reusing builds. Too difficult to parse the output, doesn't have anything to do with how it is deciding to pick the ref.
• Determine if removing the UM7 package and reinstalling successfully picks up the changes (which would mean that spack is preferring an already built artefact over a potential code change). Removing the package is not feasible as it is a dependency of other environments
• Determining if clearing the stage/buildcache is enough to get spack to pick up changes. Nope
• Other things as they come up...

[CodeGat]

Raise whether spack keeps track of the commit hash in a @git.branch reference. If it doesn't, then maybe it doesn't keep track of the state of the branch at all. Note that we couldn't find it when interrogating spack python

[CodeGat]

After chatting with a spack dev, this is potentially expected behavior, but it is behavior that they would like to change. This means that currently, @git.BRANCH references do NOT interrogate the state of the branch during concretization, and therefore they are not suitable for our use case.

We should use either a @git.TAG or @git.COMMIT reference from now on.

[CodeGat]

Linking spack#44319

[dougiesquire]

I've been playing around with Spack v1.0 to see whether it fixes this issue.

The issue

The specific situation I've been exploring/testing is this:

• The model component packages in ACCESS-OM3 all include a preferred "stable" infinity version that builds from the default branch
• In the MOM6 build-CI, I would like to build ACCESS-OM3 using the latest code on the default branch for all non-MOM6 model component packages, hence I use the default "stable" version.
• With Spack v0.22, Spack will reuse any preexisting "stable" version from the build cache, even if that was built from an older commit on the default branch.

The hope was that moving to Spack v1.0 would fix this problem since it introduces a commit=XXX reserved variant. However, that has not been my experience - see here for the PR where I have been testing.

To summarise, with Spack v1.0, when the default branch is changed, the commit variant changes in the concretized output but the spack hash stays the same and package reuse with the old commit still occurs. E.g. this is a snippet of concretization output either side of a change to the default branch:

Before a change to the default branch:

  -   hdbt3sf          ^access3-share@develop~ipo~openmp build_system=cmake build_type=Release commit=f2f35ce5915e82a83899b69560d826deab53b668 generator=make arch=linux-rocky8-x86_64_v3 %c,fortran=oneapi@2025.2.0

After a change to the default branch:

  -   hdbt3sf          ^access3-share@develop~ipo~openmp build_system=cmake build_type=Release commit=33c5bd243a7ca2263fabd1330429816de4ff2426 generator=make arch=linux-rocky8-x86_64_v3 %c,fortran=oneapi@2025.2.0

Other things I've tried

• I've tried adding no_cache=True to the version("stable",...) directive as described here. This did not seem to help.

• I've tried adding:

      reuse:
        roots: false
        from:
        - type: buildcache
          exclude:
          - 'access3'
          - 'access-cice'
          - 'access3-share'
          - 'access-generic-tracers'
          - 'access-mocsy'
          - 'access-ww3'
  

  to the spack.yaml as described here (and suggested by @anton-seaice). Again, this did not seem to help.