lfric-spack

The purpose of this repository is to install a spack environment on Archer2 to enable building and running of the Met. Office LFRic model.

Installing lfric spack packages and dependencies

Step 1 This repository uses software installed from the Met. Office simit-spack repository (https://github.com/MetOffice/simit-spack). This is a private repository so to install it you will need to get access permission from the Met. Office. By default it is assumed the simit-spack repository will be cloned into the directory above the directory containing this repository. The location of the simit-spack spack repository is specified by the SPACK_SIMIT_SPACK_REPOS environment variable. This variable is defined in the module definition file modules/lfric-spack/0.23.0.lua and can be changed if you decide to use a non-default location for the simit-spack repository.

Step 2 is to install the needed packages under a local user account, as the Archer2 login nodes do not have enough resources to build some of the packages the installation should be done using a batch job. There is an example batch job script included archer2_install.job. This job is setup to build all packages using the Gnu and Cray compiler and can be modified as needed. Run the job

sbatch archer2_install.job

The installation of rose-picker will fail (and the packages which depend on it) as it needs to fetch the source code from the Met. Office Science Repository Service, for which a valid account is needed and a password needs to be provided. It is not possible to provide the password within a batch job so this package will need to be installed on the command line

spack install rose-picker %gcc@11
spack install rose-picker %cce@15

Next resubmit archer2_install.job to install the packages which depend on rose-picker.

Step 3 is to install the packages under the LFRic shared account. First login to the lfric user account and copy file sync_lfric_shared.sh to the directory where your want the lfric spack packages to be installed. This script uses rsync to copy the lfric-spack github directory and it will also copy the Met. Office simit-spack repository. The script uses environment variables to specify the source and destination directories, change these as necessary and also change the same variables in script update_lfric_shared.sh. Run the script

./sync_lfric_shared.sh

The next step is to install the same packages as before, when installing previously a build cache was created and this can now be used to install the packages under the LFRic shared account. Installing using the build cache should be easier and quicker than before, a batch job is not necessary and no password should be needed. The install process should take less than 30 minutes. Run the script

./update_lfric_shared.sh

Update lfric spack packages when a new version of lfric_apps is released

When a new version of lfric_apps is released it is a good idea to update the simit-spack repository installed in step 1, there may be updated or new software packages needed by the new lfric_apps release. Next file envs/spack.yaml will need to be edited to add the new lfric_apps and lfric_meta package version number. Next the package.py files for lfric_apps (repos/lfric/packages/lfric-apps/package.py) and lfric_meta (repos/lfric/packages/lfric-meta/package.py) will need to be edited. Firstly add the new version number, then update any dependent software version number changes and add any new software dependencies if needed. Finally repeat steps 2 and 3 above to update all the lfric spack packages.

Using lfric spack packages

To load the environment to build and run lfric apps version 2.2, run the following for Cray compilers

module use /work/y07/shared/lfric/software/spack/lfric-spack/modules
module load PrgEnv-cray
module load lfric-spack
spack load lfric-apps@2.2%cce shumlib@13.9+openmp%cce

and the following for Gnu compilers

module use /work/y07/shared/lfric/software/spack/lfric-spack/modules
module load PrgEnv-gnu
module load lfric-spack
spack load lfric-apps@2.2%gcc shumlib@13.9+openmp%gcc

There are also lfric apps versions 2.1, 2.0 and 1.2 available (1.2 only available for Gnu compilers). Alternatively there is also the lfric-meta package which is the same as the lfric-apps package except XIOS is not loaded.

Alternatively you can load the needed packages explicitly and set some environment variables needed by the LFRic build system.

module use /work/y07/shared/lfric/software/spack/lfric-spack/modules
module load PrgEnv-gnu
module load lfric-spack
spack load yaxt%gcc py-jinja2%gcc rose-picker%gcc xios@2701%gcc py-psyclone@3.1.0%gcc shumlib@13.9+openmp%gcc
export FC=ftn
export LDMPI=ftn
export FPP="cpp -traditional-cpp"
export LFRIC_TARGET_PLATFORM=meto-xc40

Possible problems

The following error message was encountered when a user was trying to build LFRic using the Cray compiler and a pre-release version of the lfric spack packages:

ftn-3013 ftn: LIMIT LFRIC_XIOS_INITIALISE, File = lfric_xios_driver_mod.f90, Line = 1 
  Cannot open file "/tmp/jwc/spack-stage/spack-stage-xios-2701-kj5xkosn3yu2vxtemumacvzzg2fzmad5/spack-src/inc/IDATA.mod" for "rb" in file "/home/jenkins/crayftn/inl/sources/m_utils.c" at line 107.
  Reason: "Permission denied".
Command exited with non-zero status 1

The error seems to be caused by the Cray Fortran compiler hard coding a path (/tmp/jwc/spack-stage/...), from an intermediate build step, into a Fortran module file for XIOS. The compiler then complains as it doesn't have access permission to read that directory for anybody other than the person who did the installation. As a workaround you can delete the temporary spack-stage directory (rm -rf /tmp/jwc/spack-stage in this case), after installtion and the compiler no longer complains.