Problems with EPEL 10

[adrianreber]

I am currently trying to port the OpenCHAMI OpenHPC recipe to EL10 and I have problems installing EPEL 10 packages. The way EPEL 10 works is a bit different than any other repositories concerning the minor numbering. Currently I get:

ERROR - Errors during downloading metadata for repository 'epel':
ERROR -   - Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel${releasever_minor:+-z}-10&arch=x86_64&countme=1 (IP: 10.241.58.130)
ERROR -   - Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel${releasever_minor:+-z}-10&arch=x86_64 (IP: 10.241.58.130)
ERROR - Error: Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel${releasever_minor:+-z}-10&arch=x86_64 (IP: 10.241.58.130)
ERROR - Error installing packages: Command '['dnf', '--setopt=reposdir=/home/builder/.local/share/containers/storage/overlay/b7be70d404de150e3c2945b708f0f4d054c744d02f01917502c416b406081fd5/merged/etc/yum.repos.d', '--setopt=logdir=/var/tmp/image-build-_zgb5d60/dnf/log', '--setopt=cachedir=/var/tmp/image-build-_zgb5d60/dnf/cache', 'install', '-y', '--nogpgcheck', '--installroot', '/home/builder/.local/share/containers/storage/overlay/b7be70d404de150e3c2945b708f0f4d054c744d02f01917502c416b406081fd5/merged', 'ohpc-base-compute', 'ohpc-slurm-client', 'pdsh-ohpc', 'lmod-ohpc', 'nhc-ohpc']' returned non-zero exit status 1.

Is image-builder known to work with EPEL 10? Is there some setting to make it work?

[travisbcotton]

Can you post the config file you are using?

[adrianreber]

Take a look at the PR openhpc/ohpc#2245, it should have all the information. Not true, I was actually installing the epel-release package in addition. I think if you install the epel-release package as a way to bring in the repository definitions then you get the repo definition with the ${releasever_minor:+-z} magic.

Currently I switched to this:

[sms](*\#*)   - alias: 'Epel10'
[sms](*\#*)     url: 'https://dl.fedoraproject.org/pub/epel/10/Everything/x86_64/'
[sms](*\#*)     gpg: 'https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-10'

This is problematic, however, this points to EPEL 10.1 which will break some packages. epel/10 is currently a link to the upcoming 10.1 release (which matches CentOS Stream 10). So for something using a released version of RHEL 10 (or a clone) this should probably be /10z. Installing the epel-release package seems to be easier, however, and that is what I tried.

It seems, the real problem is that your builder cannot handle ${releasever_minor:+-z} which can be worked around, but it would be nice to have it work out of the box.

Looking at an EPEL mirror I see:

lrwxrwxrwx.  1 ftpadmin ftp             4 Aug 25 23:11 10 -> 10.2
drwxr-sr-x.  3 ftpadmin ftp            32 Feb 18  2025 10.0
lrwxrwxrwx.  1 ftpadmin ftp             4 Apr  4  2025 10.0z -> 10.0
drwxr-sr-x.  3 ftpadmin ftp            56 Nov  7  2024 10.1
lrwxrwxrwx.  1 ftpadmin ftp             4 Apr  4  2025 10.1z -> 10.1
drwxr-xr-x.  3 ftpadmin ftp            32 Aug 25 19:52 10.2
lrwxrwxrwx.  1 ftpadmin ftp             4 Aug 25 23:12 10.2z -> 10.2
lrwxrwxrwx.  1 ftpadmin ftp             4 Apr  4  2025 10z -> 10.0

So 10 is already pointing at 10.2. Anyway, for OpenHPC I can work around it by not installing epel-release and using /10z in the repository definition, but at some point people will try it again.

Just FYI, to make it work with EL10 I also have to run dracut with DRACUT_NO_XATTR=1 or else not all files are directly copied into the ramdisk.

[travisbcotton]

I think using ghcr.io/openchami/image-build:v0.1.2 might fix it not resolving ${releasever_minor:+-z}.
This version of the image-build tool will run the package manager commands directly in the target container so long as the parent is NOT scratch. When the parent is scratch it will do the original dnf --installroot

[adrianreber]

I think using ghcr.io/openchami/image-build:v0.1.2 might fix it not resolving ${releasever_minor:+-z}. This version of the image-build tool will run the package manager commands directly in the target container so long as the parent is NOT scratch. When the parent is scratch it will do the original dnf --installroot

This sounds almost right for OpenHPC. What I am currently doing, to not have documentation on my compute node is the following: https://github.com/openhpc/ohpc/blob/3.x/docs/recipes/install/common/openchami_base_os_build.tex#L23

Before running any package installation I am installing a file in the container which tells RPM to not install any documentation: echo -e "%_excludedocs 1" >> /opt/ohpc/admin/images/data/rpmmacros. Unfortunately after that the container is no longer FROM scratch but also does not have dnf installed in it yet. Because I also want to install dnf without documentation. Do you have any recommendation how I can use your new image-builder while keeping my %_excludedocs 1? What step happens first? Package installation or file copy? Can the order be influenced how image-builder handles these steps?

[travisbcotton]

Sorry @adrianreber, haven't had much time lately to work on this.
I've added a force_installroot option to the config file which I think will do what you want (see #48).
You can try it out by adding

options:
...
  force_installroot: true
...

To the config file.

The test container is ghcr.io/openchami/image-build:pr-48.
Lemme know if this works for you.