[BUG] Unable to build compute image with image-build:v0.1.2 on the aarch64 node

[balajikumaran-c-s]

Describe the bug
Unable to build compute image using image-build:v0.1.2 on the aarch64 node. The build fails during package installation and exits with a non-zero status. [ERROR - warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.]

To Reproduce
Steps to reproduce the behavior:
. Run the following command on an aarch64 node:
podman run --rm --device /dev/fuse --network host -e S3_ACCESS=admin -e S3_SECRET=Dell1234
--user 0 --privileged
-v /omnia/omnia/pulp/settings/certs/pulp_webserver.crt:/etc/pki/ca-trust/source/anchors/pulp_webserver.crt:z
-v /omnia/omnia/openchami/workdir/images/rhel-slurm_node_aarch64-10.0.yaml:/home/builder/config.yaml:z
--entrypoint /bin/bash ghcr.io/openchami/image-build:v0.1.2
-c "update-ca-trust extract && image-build --config /home/builder/config.yaml --log-level DEBUG"

Expected behavior
The compute image should build successfully using image-build without errors.

Screenshots

Desktop (please complete the following information):

• OS: Rhel10_aarch64

Additional context
I cloned the GitHub repository for the image builder and built the image locally using podman build. Using this locally built image, I am able to build the compute image successfully without any errors. However, when I pull the pre-built image directly from ghcr.io, the build fails with below error
ERROR - warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.

PFA for reference:
working_image output:
custom_ochami.out.txt

error_image_output
ochami_image.error.txt

[travisbcotton]

Did you update the Dockerfile when you built it locally?
I would expect it to have the same error building locally as pulling the v0.1.2 version.

The problem is that the current Dockerfile pulls in an AlmaLinux:8 image and the image-build tool essentially does a dnf --installroot... when building from scratch. RHEL8 and friends use the BDB format for the RPM database where RHEL9+ use sqlite.

When you build your base image you can try adding this to the config file:

cmds:
  - cmd: |
      rm -f /var/lib/rpm/__db*
      rpmdb --rebuilddb

That should delete the RPM database and rebuild it, hopefully with correct database type

[balajikumaran-c-s]

@travisbcotton I didn’t change any Dockerfile or its contents.
I used the same Dockerfile from the link below to build the image:
https://github.com/OpenCHAMI/image-builder/blob/main/dockerfiles/dnf/Dockerfile

I also tried building the base image using the v0.1.2 image with the command you provided. After that, I didn’t get the BDB error while building the compute image. However, I’m still unable to build the compute image because it’s throwing the following error:

**ERROR - Problem repository: [<internal_repo>pub_Distros_RedHat_RHEL10_10.0_AppStream_aarch64_os]
ERROR - bandwidth: 0
ERROR - baseurl: http://<internal_repo>/pub/Distros/RedHat/RHEL10/10.0/AppStream/aarch64/os/
ERROR - cachedir: /var/cache/dnf
ERROR - cost: 1000
ERROR - countme: 0
ERROR - deltarpm: 0
ERROR - deltarpm_percentage: 75
ERROR - enabled: 1
ERROR - enabled_metadata:
ERROR - enablegroups: 1
ERROR - exclude:
ERROR - excludepkgs:
ERROR - fastestmirror: 0
ERROR - gpgcheck: 1
ERROR - gpgkey:
ERROR - includepkgs:
ERROR - ip_resolve: whatever
ERROR - max_downloads_per_mirror: 3
ERROR - max_parallel_downloads: 3
ERROR - mediaid:
ERROR - metadata_expire: 172800
ERROR - metalink:
ERROR - minrate: 1000
ERROR - mirrorlist:
ERROR - module_hotfixes: 0
ERROR - name: created by dnf config-manager from http://<internal_repo>/pub/Distros/RedHat/RHEL10/10.0/AppStream/aarch64/os/
ERROR - password:
ERROR - priority: 99
ERROR - protected_packages: dnf, systemd, systemd-udev, grub2-tools-minimal, yum, selinux-policy-targeted, setup, redhat-release, sudo, systemd, systemd-udev, grub2-tools-minimal, yum, selinux-policy-targeted, setup, redhat-release, sudo
ERROR - proxy:
ERROR - proxy_auth_method: any
ERROR - proxy_password:
ERROR - proxy_sslcacert:
ERROR - proxy_sslclientcert:
ERROR - proxy_sslclientkey:
ERROR - proxy_sslverify: 1
ERROR - proxy_username:
ERROR - repo_gpgcheck: 0
ERROR - retries: 10
ERROR - skip_if_unavailable: 0
ERROR - sslcacert:
ERROR - sslclientcert:
ERROR - sslclientkey:
ERROR - sslverify: 1
ERROR - sslverifystatus: 0
ERROR - throttle: 0
ERROR - timeout: 30
ERROR - type:
ERROR - user_agent: libdnf (Red Hat Enterprise Linux 10.0; generic; Linux.aarch64)
ERROR - username:
ERROR - subprocess exited with status 1
ERROR - subprocess exited with status 1
ERROR - Error: exit status 1
ERROR - Error installing packages: Command '['buildah', 'run', 'rhel-slurm_node_aarch6420251106072347', '--', 'bash', '-c', 'dnf install -y munge munge-devel firewalld python3-firewall nfs-utils nfs4-acl-tools openldap-clients nss-pam-ldapd sssd oddjob-mkhomedir authselect']' returned non-zero exit status 1.
INFO - 1d92953d66e3f1d0e4711aaddf712ba1b3b2bafa72320fb25db49a47e10dcadf

-------------------BUILD LAYER--------------------
Exiting now ...
[root@chat ~]#
**

I also saw this same error earlier when I built the image without your suggested command.
However, when I built the image using the Dockerfile from the link I shared, I didn’t face this issue.

I’ve attached the complete build log below for your reference.

compute_image_error.txt

[priti-parate]

Can anyone help to resolve this? we are looking to get working flow with by this month end, please help to achieve the target

[alexlovelltroy]

Based on the information provided, this looks like a usage error rather than an error with the tool. Some things to check:

1. The error indicates a problem accessing a repository. Have you traced if this is an error with the path to the repo? the signatures of the repo? the entitlements of the repo?
2. in the repos section of your yaml file, do you have a valid gpg line for each repo and can you confirm that it works? See this example stanza
3. Have you ruled out a problem with the RHEL registration API? This tool doesn't have any way of registering a build container with a RHEL license service. If you'd like to offer a PR to include that, I'm sure other users would be interested