Intel make/cmake tests failing

[clark-evans]

The Intel make/cmake tests have begun failing for all PRs. It appears that this is tied to changes in the upstream NOAA-EMC/ci-install-intel-toolkit repository, into which several PRs and several additional commits were merged about three weeks ago. Notable among these changes is a change to the default oneAPI compiler, which is now 2025.3 rather than 2023.2.1.

The oneAPI 2025.3 compilers have gcc-13 and gfortran-13 dependencies. When the CI workflow requests that these packages be installed, however, the job fails because the requested packages are not available:

E: Unable to locate package gcc-13
E: Unable to locate package gfortran-13

See, e.g., the latest runner log file for #201, though it appears in the latest log files for #192 as well.

Our runners use Ubuntu 22.04 (see build_mpas_intel.yml, line 9). However, the Ubuntu developers documentation indicates that GCC v12 is the latest available for Ubuntu 22.04. The NOAA-EMC/ci-install-intel-toolkit appears to assume ubuntu-latest for the runners, which appears to be Ubuntu 24.04.

I think we have three potential paths forward:

• Override the upstream oneAPI default to 2023.2.1. Reverts our CI behavior to what it was a few weeks ago but means we are not taking advantage of testing with the latest compilers.
• Update our CI workflow to use Ubuntu 24.04. Carries some risk of breaking other aspects of the CI workflow, but does include GCC v12 and v13 (and v14 for that matter). Would allow us to take advantage of testing with the latest compilers for as long as we continue to use the EMC CI toolkit.
• Update our CI workflow to disable the Intel compile tests while awaiting the new containerized CI workflow that @dustinswales is developing. Not ideal unless those will be ready ASAP.

I was tempted to try option 2 myself but decided it is best to wait until talking with Dustin (as the CI developer and maintainer).

[dustinswales]

@clark-evans I tried switching to using ubuntu-latest (24.04) and the Intel toolkit installs fine, but there are other issues downstream with PNetcdf.
Looking at the logs, there seems to be other things going on in the EMC toolkit repository beyond switching OneAPI versions, which makes figuring it out a pain.
I'm reluctant to put too much effort into this as I have plans for a new containerized CI system, but that is not ready yet.

As a temporary measure, I created a personal fork of the EMC Intel toolkit repository and a new branch, ubuntu20p04, and I rolled back the recent development that was added. This is analogous to your first path forward. This is running now, and if it the test pass, I could open a PR. Thoughts?

[clark-evans]

@clark-evans I tried switching to using ubuntu-latest (24.04) and the Intel toolkit installs fine, but there are other issues downstream with PNetcdf. Looking at the logs, there seems to be other things going on in the EMC toolkit repository beyond switching OneAPI versions, which makes figuring it out a pain. I'm reluctant to put too much effort into this as I have plans for a new containerized CI system, but that is not ready yet.

As a temporary measure, I created a personal fork of the EMC Intel toolkit repository and a new branch, ubuntu20p04, and I rolled back the recent development that was added. This is analogous to your first path forward. This is running now, and if it the test pass, I could open a PR. Thoughts?

Thanks for looking into this, @dustinswales. I'm wondering if, instead of creating a personal fork of the EMC Intel toolkit repo, we could modify the build_mpas_intel.yml workflow driver to override the defaults in the upstream repo? Specifically, instead of just specifying "with: install-mpi: true," we could add "oneapi-version: '2023.2.1'" (following the specifications in the upstream actions.yml), though we may also need to override the compiler names (ifx -> ifort) in a similar fashion?

That said, if you think it's easier to fork and adapt to that fork, I'm OK with that. I concur on not spending a lot of effort on this given your plans for a new containerized CI system.

[dustinswales]

@clark-evans This one line change gets the tests working again.

Maybe there is a way we can specify the commit hash we use in NOAA-EMC/ci-install-intel-toolkit?
Replace:
uses: NOAA-EMC/ci-install-intel-toolkit@develop
With something like:
uses: NOAA-EMC/ci-install-intel-toolkit@develop:35f982eddbc087b6c3c74151f6d32a177cee7812

[clark-evans]

Fixed by #208.