Initial software: Myriad packages

[heatherkellyucl]

From #44

Samtools, BCFTools, HTSlib: UCL-ARC/rcps-buildscripts#532

Samtools, Bwa, Bedtools, Vcftools, picard, GATK (requested by Biosciences) https://ucldata.atlassian.net/browse/AHS-139

Alphafold (would be nice to not have to tell people to go and track down the working containers): UCL-ARC/rcps-buildscripts#529 UCL-ARC/rcps-buildscripts#463

HDF5
Netcdf
BEAST 2.7: UCL-ARC/rcps-buildscripts#498

[heatherkellyucl]

On Myriad, continuing from existing site with compilers only installed:

source spacksites/myriad-utilities/init-spacksites-on-myriad.sh
eval $(spacksites/spacksites spack-setup-env hk-initial-stack)

Looking at adding these to the environment

htslib@1.16 %gcc@12.2.0
samtools@1.16.1 %gcc@12.2.0
bcftools@1.16 %gcc@12.2.0

(want them all to be 1.16 or we get two htslibs, since samtools 1.16.1 wants htslib 1.16).

[heatherkellyucl]

Fun times, ran into spack/spack#35197 with gzip as an htslib dependency.

Works if you get it to use gzip 1.13 instead with spack checksum --add-to-package gzip (which now should be the one in use as everything less has a vulnerability and needs to be deprecated - To Be Pull Requested).

[heatherkellyucl]

gzip is updated in develop branch (https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/gzip/package.py). Need to work out what exactly we do to make sure we include that version here when creating a new environment. Add to local specs?

[giordano]

Since gzip 1.12- is deprecated, the resolver should basically only consider gzip@1.13 (unless you explicitly use --deprecated. If you want to be super explicit, in your config you could have

packages:
  gzip:
    require: "@1.13:"

[heatherkellyucl]

It is only deprecated in the develop branch though - so we won't see that in 0.20 unless we do something to make sure we get that version of the package.py.

[heatherkellyucl]

(We'll get https://github.com/spack/spack/blob/releases/v0.20/var/spack/repos/builtin/packages/gzip/package.py)

[giordano]

So you wouldn't have gzip@1.13 either 😞

[heatherkellyucl]

Yeah. We will need to solve this in the more general case for anything else where we need to alter the package.py for whatever reason. Which is why I was thinking of a local package location too as part of this repo, where new packages and altered packages go.

[heatherkellyucl]

repos.yaml: https://spack.readthedocs.io/en/latest/repositories.html

[heatherkellyucl]

It'll look through the repos in order, so we put ours first and it'll get gzip from there.

On updating the version of Spack we are pinned to, we would need to check whether we should remove packages from our own repo to return to using Spack's ones.

[heatherkellyucl]

bcftools@1.16 depends on python and at least a few packages :sadface:

Didn't want to mess with python yet, might have to.

Actually, this might be fine, we just wouldn't have created our view and associated module yet.

[heatherkellyucl]

James started a local repo as repos/dev in the top level in here. We probably want a repos/ucl too for when things are non-dev.

[heatherkellyucl]

Got these after installing all three:

-- linux-rhel7-skylake_avx512 / gcc@12.2.0 ----------------------
bcftools@1.16                       libxml2@2.10.3        py-pip@23.0
berkeley-db@18.1.40                 nasm@2.15.05          py-pybind11@2.10.1
bzip2@1.0.8                         ncurses@6.4           py-pyparsing@3.0.9
ca-certificates-mozilla@2023-01-10  ninja@1.11.1          py-pyproject-hooks@1.0.0
cmake@3.26.3                        openblas@0.3.23       py-python-dateutil@2.8.2
curl@8.0.1                          openssl@1.1.1t        py-setuptools@63.4.3
diffutils@3.9                       perl@5.36.0           py-setuptools-scm@7.0.5
expat@2.5.0                         pigz@2.7              py-six@1.16.0
freetype@2.11.1                     pkgconf@1.9.5         py-tomli@2.0.1
gdbm@1.23                           py-build@0.10.0       py-typing-extensions@4.5.0
gettext@0.21.1                      py-certifi@2022.12.7  py-wheel@0.37.1
gmake@4.4.1                         py-contourpy@1.0.5    python@3.10.10
gzip@1.13                           py-cppy@1.2.1         qhull@2020.2
htslib@1.16                         py-cycler@0.11.0      re2c@2.2
libbsd@0.11.7                       py-cython@0.29.33     readline@8.2
libdeflate@1.10                     py-flit-core@3.7.1    samtools@1.16.1
libffi@3.4.4                        py-fonttools@4.37.3   sqlite@3.40.1
libiconv@1.17                       py-kiwisolver@1.4.4   tar@1.34
libjpeg-turbo@2.1.5                 py-matplotlib@3.7.1   util-linux-uuid@2.38.1
libmd@1.0.4                         py-numpy@1.24.3       xz@5.4.1
libpng@1.6.39                       py-packaging@23.0     zlib@1.2.13
libxcrypt@4.4.33                    py-pillow@9.5.0       zstd@1.5.5

[heatherkellyucl]

We now have these in this repo:
https://github.com/UCL-ARC/hpc-spack/blob/0.20/repos/ucl/repo.yaml
https://github.com/UCL-ARC/hpc-spack/blob/0.20/repos/ucl/packages/gzip/package.py