Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[submodule "spack"]
path = spack
url = https://github.com/jcsda/spack
branch = spack-stack-1.9.2rc1
branch = release/1.9.0
[submodule "doc/CMakeModules"]
path = doc/CMakeModules
url = https://github.com/noaa-emc/cmakemodules
Expand Down
8 changes: 5 additions & 3 deletions configs/common/packages.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ packages:
magics:
require: "@4.15.3:"
mapl:
require: '@2.53.0 ~shared ~f2py'
require: '@2.53 ~shared ~f2py'
variants: '+pflogger'
met:
require: '@11.1.1 +python +grib2'
Expand Down Expand Up @@ -265,9 +265,11 @@ packages:
py-urllib3:
require: '@1.26.12'
qt:
require: '@5'
require:
- '@5'
scotch:
require: '@7.0.4 +mpi+metis~shared~threads~mpi_thread+noarch+esmumps'
require:
- '@7.0.4 +mpi+metis~shared~threads~mpi_thread+noarch+esmumps'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean we are using scotch 7.0.4 or are we moving to 7.0.7? (sorry this is my lack of spack-stack knowledge question)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still scotch 7.0.4. If it works, then there is no need to update it. The purpose of spack-stack release 1.9.2 is to provide a bug fix / hot fix for the existing 1.9.0 and 1.9.1 releases. Bug fix releases only contain the necessary changes to fix bugs, no changes that aren't necessary.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if scotch 7.0.4 works on other machines? Or just Ursa... I'm assuming it's going to work on other machines but don't know. Either way WW3 is ready to handle a mix of the two if necessary.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure it will work. We'll install the 1.9.2 release candidate on two machines of your choice with both GNU and oneAPI, and hopefully that will confirm my guess. Ursa + ???

sfcio:
require: '@1.4.2'
shumlib:
Expand Down
3 changes: 3 additions & 0 deletions configs/common/packages_oneapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,6 @@ packages:
qt:
require:
- '%gcc'
scotch:
require:
- '%gcc'
2 changes: 1 addition & 1 deletion configs/containers/specs/jedi-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
w3emc@2.10.0,
nco,
esmf@8.8.0,
mapl@2.53.0,
mapl@2.53.4,
zlib-ng,
zstd,
odc@1.5.2,
Expand Down
2 changes: 2 additions & 0 deletions configs/sites/tier1/ursa/compilers.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ compilers:
fc: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2024.2.1-oqhstbmawnrsdw472p4pjsopj547o6xs/compiler/2024.2/bin/ifort
flags: {}
operating_system: rocky9
target: x86_64
modules:
- intel-oneapi-compilers/2024.2.1
environment:
Expand All @@ -26,6 +27,7 @@ compilers:
fc: /usr/bin/gfortran
flags: {}
operating_system: rocky9
target: x86_64
Copy link
Collaborator Author

@climbfuji climbfuji Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target entries are the magic that fixes all the version conflicts.

modules: []
environment: {}
extra_rpaths: []
14 changes: 4 additions & 10 deletions configs/sites/tier1/ursa/packages_gcc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,8 @@ packages:
prefix: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/openmpi-4.1.6-auzmzlihoo7n6g254nqkguxtzrdzkyqv
modules:
- openmpi/4.1.6
gcc-runtime:
externals:
- spec: gcc-runtime@11.4.1%gcc@11.4.1
prefix: /usr

ectrans:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant

require::
- '~mkl +fftw'
gsibec:
require::
- '~mkl'
py-numpy:
require::
- '@1.26'
- '^openblas'
24 changes: 11 additions & 13 deletions configs/sites/tier1/ursa/packages_oneapi.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
packages:
all:
compiler:: [oneapi@2024.2.1]
compiler:: [oneapi@2024.2.1,gcc@11.4.1]
providers:
mpi:: [intel-oneapi-mpi@2021.13]
# Change as appropriate to switch to intel-oneapi-mkl
Expand All @@ -16,22 +16,20 @@ packages:
modules:
- intel-oneapi-mpi/2021.13.1
prefix: /apps/spack-2024-12/linux-rocky9-x86_64/oneapi-2024.2.1/intel-oneapi-mpi-2021.13.1-ss72gbndvat3oz22sa6lhmlbjkeabrn4

intel-oneapi-mkl:
externals:
- spec: intel-oneapi-mkl@2024.2.1%oneapi@2024.2.1
modules:
- intel-oneapi-mkl/2024.2.1
prefix: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-mkl-2024.2.1-srqwrbzwo2k7hxuuhlrxttburs5jvlat
intel-oneapi-runtime:
externals:
- spec: intel-oneapi-runtime@2024.2.1%oneapi@2024.2.1
prefix: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2024.2.1-oqhstbmawnrsdw472p4pjsopj547o6xs/compiler/2024.2
modules:
- compiler-rt/2024.2.1
gcc-runtime:
externals:
- spec: gcc-runtime@11.4.1%gcc@11.4.1
prefix: /usr

# If using intel-oneapi-mkl, make appropriate changes below
Copy link
Collaborator Author

@climbfuji climbfuji Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a new platform, let's just move straight back to Intel MKL instead of OpenBLAS+FFTW with Intel.

The GCC build still uses OpenBLAS and FFTW, as intended.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what Jessica Meixner tested in ufs-community/ufs-weather-model#2650 (comment)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@climbfuji will going "straight back to Intel MKL instead of OpenBLAS+FFTW with Intel" on ursa but not on other problems cause any inconsistencies? trying to understand if we are going to need to rebuild 1.9.1 entirely on all platforms or if we can hopefully just update scotch...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it won't. It's entirely transparent to the users or downstream application. I've asked for this move in general for a very long time. Before we had OpenBLAS with Intel, hpc-stack and jedi-stack used MKL with Intel by default. When spack-stack started, we had problems with getting MKL to work and out of necessity switched to OpenBLAS.

In fact, we already switched to MKL on all NRL systems with Intel (and use OpenBLAS with GNU), and nobody noticed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spack-stack CI runners are like that, too, by the way.

ectrans:
require::
- '~mkl +fftw'
gsibec:
require::
- '~mkl'
py-numpy:
require::
- '@1.26'
- '^openblas'
2 changes: 1 addition & 1 deletion configs/templates/skylab-dev/spack.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spack:

# Mapl with various esmf tags
- mapl@2.53.0 ^esmf@8.6.1
- mapl@2.53.0 ^esmf@8.8.0
- mapl@2.53.4 ^esmf@8.8.0

# Various esmf tags (list all to avoid duplicate packages)
- esmf@=8.6.1 snapshot=none
Expand Down
2 changes: 1 addition & 1 deletion configs/templates/unified-dev/spack.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ spack:

# Mapl with various esmf tags
- mapl@2.53.0 ^esmf@8.6.1
- mapl@2.53.0 ^esmf@8.8.0
- mapl@2.53.4 ^esmf@8.8.0

# Various esmf tags (list all to avoid duplicate packages)
- esmf@=8.6.1 snapshot=none
Expand Down
Loading