ci: re-enable darwin stacks post machine upgrade

[eugeneswalker]

CC @kwryankrattiger

[zackgalbreath]

It tooks like we have these stacks disabled both within .gitlab-ci.yml and within the GitLab web UI. I'm going to run a new pipeline for this branch with the darwin stacks re-enabled now to verify if they still build successfully. If not, I would argue that we should also fix any regressions in these stacks as part of this PR.

[zackgalbreath]

Looks like we'll need to update some tags too?

[zackgalbreath]

need to update tags as well, see screenshot

[eugeneswalker]

need to update tags as well, see screenshot

Just pushed this update. Thanks for pointing this out.

[eugeneswalker]

Our Tahoe upgrade seems to be a problem:

• ml-darwin-aarch64-mps-generate

...
$ spack -v  --color=always ci generate --check-index-only -j ${SPACK_CONCRETIZE_JOBS} --forward-variable SPACK_CHECKOUT_VERSION --forward-variable SPACK_CHECKOUT_REPO --forward-variable SPACK_CI_PACKAGES_ROOT --forward-variable SPACK_CI_SPACK_ROOT --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir/${SPACK_CI_STACK_NAME}" --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/${SPACK_CI_STACK_NAME}/cloud-ci-pipeline.yml"
...
Traceback (most recent call last):
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/bin/spack", line 49, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/main.py", line 1123, in main
    return _main(argv)
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/main.py", line 1075, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/main.py", line 1106, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/main.py", line 649, in _invoke_command
    return_val = command(parser, args)
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/cmd/ci.py", line 891, in ci
    return args.func(args)
           ~~~~~~~~~^^^^^^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/cmd/ci.py", line 257, in ci_generate
    spack_ci.generate_pipeline(env, args)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/ci/__init__.py", line 477, in generate_pipeline
    env.concretize()
    ~~~~~~~~~~~~~~^^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/environment/environment.py", line 1566, in concretize
    return self._concretize_separately(tests=tests)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/environment/environment.py", line 1716, in _concretize_separately
    concretized_specs = spack.concretize.concretize_separately(to_concretize, tests=tests)
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/concretize.py", line 157, in concretize_separately
    for j, (i, concrete, duration) in enumerate(
                                      ~~~~~~~~~^
        spack.util.parallel.imap_unordered(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            _concretize_task, args, processes=num_procs, debug=tty.is_debug(), maxtaskperchild=1
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
    ):
    ^
  File "/Users/gitlab-runner-p-1/builds/t1_ZipHk_/0/spack/spack-packages/.ci/tmp/spack/lib/spack/spack/util/parallel.py", line 92, in imap_unordered
    raise RuntimeError(result.stacktrace if debug else str(result))
RuntimeError: failed to concretize `py-tensorflow-probability` for the following reasons:
     1. bazel: 'os=tahoe' conflicts with '@:7.6.1,8:8.4.1'
     2. bazel: 'os=tahoe' conflicts with '@:7.6,8:8.4.1'
     3. bazel: 'os=tahoe' conflicts with '@:7.6.1,8:8.4.1'
        required because conflict constraint @:7.6.1,8:8.4.1 
          required because py-tensorflow-probability depends on bazel@3.2:6 
            required because py-tensorflow-probability requested explicitly 
        required because conflict is triggered when os=tahoe 
          required because py-tensorflow-probability depends on bazel@3.2:6 
            required because py-tensorflow-probability requested explicitly 
     4. bazel: 'os=tahoe' conflicts with '@:7.6,8:8.4.1'
        required because conflict is triggered when os=tahoe 
          required because py-tensorflow-probability depends on bazel@3.2:6 
            required because py-tensorflow-probability requested explicitly 
        required because conflict constraint @:7.6,8:8.4.1 
          required because py-tensorflow-probability depends on bazel@3.2:6 
            required because py-tensorflow-probability requested explicitly 

[zackgalbreath]

running another manual pipeline for this branch because the mac stacks are still double disabled

[eugeneswalker]

All the darwin jobs, irrespective of stack, are failing with these type of errors:

• gcc-runtime@14.2.0 /wzh4niz4piw3s4u5iwnzrt3kbop3h5or Machine Learning MPS

...
$ echo "$REMOTE_SCRIPT_CLONE_SPACK_SHA256  clone_spack.sh" | sha256sum -c
usage: sha256sum [-bctwz] [files ...]
Running after_script 00:00
Uploading artifacts for failed job 00:00
Uploading artifacts...
Runtime platform                                    arch=arm64 os=darwin pid=60250 revision=374d34fd version=17.6.0
WARNING: jobs_scratch_dir/ml-darwin-aarch64-mps/logs: no matching files. Ensure that the artifact path is relative to the working directory (/Users/gitlab-runner-1/builds/t1_4RcQ4y/0/spack/spack-packages) 
WARNING: jobs_scratch_dir/ml-darwin-aarch64-mps/reproduction: no matching files. Ensure that the artifact path is relative to the working directory (/Users/gitlab-runner-1/builds/t1_4RcQ4y/0/spack/spack-packages) 
WARNING: jobs_scratch_dir/ml-darwin-aarch64-mps/tests: no matching files. Ensure that the artifact path is relative to the working directory (/Users/gitlab-runner-1/builds/t1_4RcQ4y/0/spack/spack-packages) 
WARNING: jobs_scratch_dir/ml-darwin-aarch64-mps/user_data: no matching files. Ensure that the artifact path is relative to the working directory (/Users/gitlab-runner-1/builds/t1_4RcQ4y/0/spack/spack-packages) 
ERROR: No files to upload            

• ca-certificates-mozilla@2025-08-12 /67uaxjrjrmnlenpxvvnkwmsl6gkf6wk3 Machine Learning MPS

...
$ time ./spack/bin/spack python ${SPACK_CI_PACKAGES_ROOT}/.ci/gitlab/scripts/common/aggregate_package_logs.spack.py --prefix /home/software/spack:${CI_PROJECT_DIR}/opt/spack --log install_times.json ${SPACK_ARTIFACTS_ROOT}/user_data/install_times.json || true
bash: line 215: ./spack/bin/spack: No such file or directory
real	0m0.002s
user	0m0.000s
sys	0m0.001s
Uploading artifacts for failed job 00:00
Uploading artifacts...
Runtime platform                                    arch=arm64 os=darwin pid=51748 revision=374d34fd version=17.6.0
...

[eugeneswalker]

Looks like we need to rebase and make sure this fix is included:

• ci: remove sha256 long args #2957

Doing that next...

[eugeneswalker]

@zackgalbreath Can you re-trigger manual run with the right variables?

[zackgalbreath]

@zackgalbreath Can you re-trigger manual run with the right variables?

Didn't work unfortunately. It looks like the problem is that the macOS version of sha256sum -c can't read from stdin. I'll push a fix shortly...

[zackgalbreath]

We got a little bit further that time. New error is:

KeyError: 'tahoe'

at lib/spack/spack/platforms/darwin.py#L47

[zackgalbreath]

KeyError: 'tahoe'

This looks like the relevant issue: spack/spack#51589

[zackgalbreath]

Upon closer inspection, I think the issue is that rho is tagged as tahoe but appears to be still running sequoia. For now I've paused the rho runners and kicked off another pipeline.

[zackgalbreath]

We're getting a lot further this time, but I noticed some errors related to gfortran.

[eugeneswalker]

We're getting a lot further this time, but I noticed some errors related to gfortran.

The issue seems to be that the GNU Fortran compiler version we are using is incompatible with apple clang post upgrade. I am working on this, will post update here.

[trws]

How does this play with #3144? They both seem to touch a lot of the same spots.

[adamjstewart]

@spackbot run pipeline

[spackbot-app]

I've started that pipeline for you!

[adamjstewart]

How does this play with #3144? They both seem to touch a lot of the same spots.

#3144 should be merged after this PR. This PR re-enables CI, #3144 reorganizes that CI.

[adamjstewart]

#3119 is now merged, you should be able to rebase to get CI finally passing!