[Owain testing] Automated builds on Condenser

[owainkenwayucl]

One of the side-projects I think might be usuful is having an automated process do some test builds on Condenser on the latest EL 9.x so we can know ahead of time what breaks.

I'm happy to start working on this with the idea that eventually we'll have a Terraform + Ansible/plan + playbook which can deploy "UCL HPC stack look-alike things" onto the Cloud infrastructure.

(note previous attempts to build things here: #61)

This should also result in "more automated" setup. I hope.

[heatherkellyucl]

Deployment procedure of Kathleen 2025-09 stack

This is for a spacksite called 2025-09 using the yamls in https://github.com/UCL-ARC/hpc-spack/tree/0.23/spacksites/spack-env-templates/slurm/2025-09 and the settings in https://github.com/UCL-ARC/hpc-spack/tree/0.23/spacksites/settings (note there are subdirectories for youngmichael which had some changes).

Potentially the buildcache commands can be totally ignored if the goal is to build everything fresh each time.

source spacksites/myriad-utilities/init-spacksites-rhel9.sh
spacksites/spacksites --config-file spacksites/settings/deploy_sites.ini create 2025-09 
# if a buildcache already existed, may need to trust its keys - not needed for a total fresh start
/lustre/scratch/ccspapp/spack/0.23/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh \
  /lustre/apps/spack/0.23/deploy/2025-09/spack/share/spack/setup-env.sh \
  spack gpg trust /lustre/apps/spack/0.23/buildcache/build_cache/_pgp/BDDF48FC04F6D2FA303270E623C30EDB42F2011F.pub
spacksites/spacksites --config-file spacksites/settings/deploy_sites.ini install-env 2025-09 compiler first_compiler.yaml
eval $(spacksites/spacksites --config-file spacksites/settings/deploy_sites.ini spack-setup-env 2025-09)

# we need a gcc 12 built with gcc 12 for annoying perl hardcoded things
spack env create second_compiler spacksites/spack-env-templates/slurm/2025-09/second_compiler.yaml
spack -e second_compiler concretize -U -f
spack -e second_compiler install --only-concrete
spack compiler remove gcc@12.3.0 
spack compiler add $(spack -e second_compiler location -i gcc@12.3.0%gcc@12.3.0)

# build envs - base env installs the intel oneapi compiler
spack env create base spacksites/spack-env-templates/slurm/2025-09/base.yaml
spack -e base concretize -U -f
spack -e base install --only-concrete

# make sure we don't keep the original perl with outdated paths (may be unnecessary for fresh build)
spack -e base install --no-cache --overwrite perl
spack -e base buildcache push /lustre/apps/spack/0.23/buildcache
spack buildcache update-index /lustre/apps/spack/0.23/buildcache
spack compiler add $(spack -e base location -i intel-oneapi-compilers)/compiler/latest/bin
# Manual edit: also need to add to environment: prepend_path: PATH: in compilers.yaml to add bin directory for fpp

spack env create ompi412 spacksites/spack-env-templates/slurm/2025-09/ompi412.yaml
spack -e ompi412 concretize -U -f
spack -e ompi412 install --only-concrete

spack env create intel spacksites/spack-env-templates/slurm/2025-09/intel.yaml
spack -e intel concretize -U -f
spack -e intel install --only-concrete

spack env create kathleen spacksites/spack-env-templates/slurm/2025-09/kathleen.yaml
spack -e kathleen concretize -U -f
spack -e kathleen install --only-concrete

# if multiple goes creating an environment were needed, the modules need recreating. Or spack -e base module tcl refresh needs running once for each env
spack -e base module tcl refresh --delete-tree -y

[heatherkellyucl]

The perl issue is that it creates a lib/5.38.0/x86_64-linux-thread-multi/Config.pm which has a setting for cc and this needs to be a compiler that is still installed as rare other perl packages sometimes try to use that for their compiler.

[owainkenwayucl]

Wrt to

# Manual edit: also need to add to environment: prepend_path: PATH: in compilers.yaml to add bin directory for fpp

https://github.com/owainkenwayucl/dev-infra/blob/main/ucl-hpc-stack-builder-environment/roles/spacksites/files/fix_compilers.py

[owainkenwayucl]

Manual success!

[ccspapp@uccaoke-spacksites-01-a485114842 ~]$ module load default-modules
Loading default-modules/2025-05
  Loading requirement: curl/8.10.1/gcc-12.3.0 cmake/3.30.5/gcc-12.3.0 git/2.46.2/gcc-12.3.0 apr/1.7.5/gcc-12.3.0 apr-util/1.6.3/gcc-12.3.0 subversion/1.14.2/gcc-12.3.0 compilers/gcc/12.3.0/gcc-12.3.0
    numactl/2.0.18/gcc-12.3.0 python/3.11.6/gcc-12.3.0 mpi/openmpi/4.1.6/gcc-12.3.0 userscripts/2025-05
[ccspapp@uccaoke-spacksites-01-a485114842 ~]$ module avail
----------------------------------------------------------------------- /apps/spack/0.23/deploy/2025-09/modules/linux-almalinux9-icelake ------------------------------------------------------------------------
apr-util/1.6.3/gcc-12.3.0                   fftw/3.3.10/gcc-12.3.0                                            lammps/20240829.1-openmpi/gcc-12.3.0                  orca/5.0.4/gcc-12.3.0                        
apr/1.7.5/gcc-12.3.0                        gatk/4.4.0.0/gcc-12.3.0                                           libxc/6.2.2/gcc-12.3.0                                orca/6.0.1/gcc-12.3.0                        
bcftools/1.16/gcc-12.3.0                    git/2.46.2/gcc-12.3.0                                             mpi/intel-oneapi-mpi/2021.14.0/intel-oneapi-2024.2.1  picard/2.26.2/gcc-12.3.0                     
beast2/2.7.4/gcc-12.3.0                     gromacs/2023.5-openmpi-plumed/gcc-12.3.0                          mpi/openmpi/4.1.2/gcc-12.3.0                          plumed/2.9.2-openmpi-cuda/gcc-12.3.0         
bedtools2/2.31.0/gcc-12.3.0                 gromacs/2024.3-openmpi/gcc-12.3.0                                 mpi/openmpi/4.1.6/gcc-12.3.0                          python/3.11.6/gcc-12.3.0                     
bwa/0.7.17/gcc-12.3.0                       gsl/2.8/gcc-12.3.0                                                mpi/openmpi/4.1.6/oneapi-2024.2.1                     quantum-espresso/7.4-openmpi/gcc-12.3.0      
castep/23.1-openmpi/gcc-12.3.0              gzip/1.13/gcc-12.3.0                                              namd/3.0.1-ucx/gcc-12.3.0                             samtools/1.17/gcc-12.3.0                     
castep/24.1-openmpi/gcc-12.3.0              hdf5/1.14.5-intel-oneapi-mpi/oneapi-2024.2.1                      netcdf-c/4.9.2/gcc-12.3.0                             subversion/1.14.2/gcc-12.3.0                 
cmake/3.30.5/gcc-12.3.0                     hdf5/1.14.5-openmpi/gcc-12.3.0                                    netcdf-fortran/4.6.1/gcc-12.3.0                       vasp/5.4.4-intel-oneapi-mpi/oneapi-2024.2.1  
compilers/gcc/12.3.0/gcc-12.3.0             hdf5/1.14.5/gcc-12.3.0                                            netlib-scalapack/2.2.0-openmpi/gcc-12.3.0             vasp/6.4.3-intel-oneapi-mpi/oneapi-2024.2.1  
compilers/intel-oneapi/2024.2.1/gcc-12.3.0  htslib/1.17/gcc-12.3.0                                            numactl/2.0.18/gcc-12.3.0                             vasp/6.5.1-intel-oneapi-mpi/oneapi-2024.2.1  
cp2k/2024.3-openmpi/gcc-12.3.0              intel-oneapi-mkl/2023.2.0-intel-oneapi-mpi/intel-oneapi-2024.2.1  openblas/0.3.28-omp/gcc-12.3.0                        
curl/8.10.1/gcc-12.3.0                      intel-oneapi-mkl/2023.2.0-openmpi/intel-oneapi-2024.2.1           openblas/0.3.28/gcc-12.3.0                            

---------------------------------------------------------------------------------------- /apps/hpc-modulefiles/kathleen -----------------------------------------------------------------------------------------
default-modules/2025-05  ucl-stack/2025-05  userscripts/2025-05  

Key:
loaded  modulepath  auto-loaded  
[ccspapp@uccaoke-spacksites-01-a485114842 ~]$ 

Now I need to see how much of the ansible I wrote actually works.

[owainkenwayucl]

Because I'm a big fan of my own suffering, I'm also looking to see if this works on AlmaLinux 10. It doesn't. If I find fixes, I'll add notes here.

The PSM library doesn't build (not looked at it yet but it's a double declaration). HDF5 doesn't build because Redhat has moved /usr/lib64/libz.a from -devel into zlib-ng-compat-static. So that's the first fix...

[owainkenwayucl]

The opa-psm2 compile error is

/lustre/apps/spack/0.23/deploy/2025-09/build_stage/ccspapp/spack-stage-opa-psm2-11.2.230-izdx5vxaubij6zh37pkaknm6ok46pwby/spack-src/ptl_ips/ips_proto_dump.c:152:15: error: static declaration of 'strlcat' follows non-static declaration
  152 | static size_t strlcat(char *d, const char *s, size_t l)
      |               ^~~~~~~
In file included from /lustre/apps/spack/0.23/deploy/2025-09/spack/opt/spack/linux-almalinux10-icelake/gcc-12.3.0/numactl-2.0.18-lodirwmyyrn7wxpy77pkrzra6kxnbgk6/include/numa.h:26,
                 from /lustre/apps/spack/0.23/deploy/2025-09/build_stage/ccspapp/spack-stage-opa-psm2-11.2.230-izdx5vxaubij6zh37pkaknm6ok46pwby/spack-src/psm_user.h:70,
                 from /lustre/apps/spack/0.23/deploy/2025-09/build_stage/ccspapp/spack-stage-opa-psm2-11.2.230-izdx5vxaubij6zh37pkaknm6ok46pwby/spack-src/ptl_ips/ips_proto_dump.c:56:
/usr/include/string.h:512:15: note: previous declaration of 'strlcat' with type 'size_t(char * restrict,  const char * restrict,  size_t)' {aka 'long unsigned int(char * restrict,  const char * restrict,  long unsigned int)'}
  512 | extern size_t strlcat (char *__restrict __dest,
      |               ^~~~~~~
make[1]: *** [Makefile:80: /lustre/apps/spack/0.23/deploy/2025-09/build_stage/ccspapp/spack-stage-opa-psm2-11.2.230-izdx5vxaubij6zh37pkaknm6ok46pwby/spack-src/build_release/ptl_ips/ips_proto_dump.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/lustre/apps/spack/0.23/deploy/2025-09/build_stage/ccspapp/spack-stage-opa-psm2-11.2.230-izdx5vxaubij6zh37pkaknm6ok46pwby/spack-src/ptl_ips'
make: *** [Makefile:301: all] Error 1

[owainkenwayucl]

I actually don't know if that library is even built on 9.x as it possesses libpsm2 which is not available as a package on 10.x.

[owainkenwayucl]

I took psm out of the openmpi config on the AlmaLinux 10.1 machine for now. I'll probably fork and create a platfrom for el10...

[owainkenwayucl]

https://github.com/owainkenwayucl/hpc-spack/tree/el10

[owainkenwayucl]

Modulo the psm stuff:

[ccspapp@uccaoke-spacksites-01-194c94e360 ~]$ cat /etc/redhat-release 
AlmaLinux release 10.1 (Heliotrope Lion)
[ccspapp@uccaoke-spacksites-01-194c94e360 ~]$ uname -a
Linux uccaoke-spacksites-01-194c94e360 6.12.0-124.43.1.el10_1.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 11 06:19:32 EDT 2026 x86_64 GNU/Linux
[ccspapp@uccaoke-spacksites-01-194c94e360 ~]$ module avail
------------------------------------------- /apps/spack/0.23/deploy/2025-09/modules/linux-almalinux10-icelake --------------------------------------------
apr-util/1.6.3/gcc-12.3.0                                         intel-oneapi-mkl/2023.2.0-openmpi/intel-oneapi-2024.2.1  
apr/1.7.5/gcc-12.3.0                                              lammps/20240829.1-openmpi/gcc-12.3.0                     
bcftools/1.16/gcc-12.3.0                                          libxc/6.2.2/gcc-12.3.0                                   
beast2/2.7.4/gcc-12.3.0                                           mpi/intel-oneapi-mpi/2021.14.0/intel-oneapi-2024.2.1     
bedtools2/2.31.0/gcc-12.3.0                                       mpi/openmpi/4.1.2/gcc-12.3.0                             
bwa/0.7.17/gcc-12.3.0                                             mpi/openmpi/4.1.6/gcc-12.3.0                             
castep/23.1-openmpi/gcc-12.3.0                                    mpi/openmpi/4.1.6/oneapi-2024.2.1                        
castep/24.1-openmpi/gcc-12.3.0                                    namd/3.0.1-ucx/gcc-12.3.0                                
cmake/3.30.5/gcc-12.3.0                                           netcdf-c/4.9.2/gcc-12.3.0                                
compilers/gcc/12.3.0/gcc-12.3.0                                   netcdf-fortran/4.6.1/gcc-12.3.0                          
compilers/intel-oneapi/2024.2.1/gcc-12.3.0                        netlib-scalapack/2.2.0-openmpi/gcc-12.3.0                
cp2k/2024.3-openmpi/gcc-12.3.0                                    numactl/2.0.18/gcc-12.3.0                                
curl/8.10.1/gcc-12.3.0                                            openblas/0.3.28-omp/gcc-12.3.0                           
fftw/3.3.10/gcc-12.3.0                                            openblas/0.3.28/gcc-12.3.0                               
gatk/4.4.0.0/gcc-12.3.0                                           orca/5.0.4/gcc-12.3.0                                    
git/2.46.2/gcc-12.3.0                                             orca/6.0.1/gcc-12.3.0                                    
gromacs/2023.5-openmpi-plumed/gcc-12.3.0                          picard/2.26.2/gcc-12.3.0                                 
gromacs/2024.3-openmpi/gcc-12.3.0                                 plumed/2.9.2-openmpi-cuda/gcc-12.3.0                     
gsl/2.8/gcc-12.3.0                                                python/3.11.6/gcc-12.3.0                                 
gzip/1.13/gcc-12.3.0                                              quantum-espresso/7.4-openmpi/gcc-12.3.0                  
hdf5/1.14.5-intel-oneapi-mpi/oneapi-2024.2.1                      samtools/1.17/gcc-12.3.0                                 
hdf5/1.14.5-openmpi/gcc-12.3.0                                    subversion/1.14.2/gcc-12.3.0                             
hdf5/1.14.5/gcc-12.3.0                                            vasp/5.4.4-intel-oneapi-mpi/oneapi-2024.2.1              
htslib/1.17/gcc-12.3.0                                            vasp/6.4.3-intel-oneapi-mpi/oneapi-2024.2.1              
intel-oneapi-mkl/2023.2.0-intel-oneapi-mpi/intel-oneapi-2024.2.1  vasp/6.5.1-intel-oneapi-mpi/oneapi-2024.2.1              

------------------------------------------------------------- /apps/hpc-modulefiles/kathleen -------------------------------------------------------------
comsol/6.4-chemeng  comsol/6.4-eee  default-modules/2025-05  star-cd/4.28.050  ucl-stack/2025-05  userscripts/2025-05  

Key:
modulepath  
[ccspapp@uccaoke-spacksites-01-194c94e360 ~]$ 

[owainkenwayucl]

OK, finally I have got the Ansible stuff to work and so now we can do automated builds of a VM with either AL 9.x or AL 10.x and the current Kathleen stack (modulo PSM for AL 10.x).

[owainkenwayucl]

Ok, I'm now working on genericising the playbooks so they automatically do the right thing on different versions of AlmaLinux (currently I have multiple playbooks).

[owainkenwayucl]

After a lot of silly problems caused by long builds and not enough caffeine I now have working "generic" spacksites ansibile role.