Image Flavors: Wrong base inheritance

[max06]

(Assuming the image definitions in .obs/dockerfile/* are used to offer the image channels for the elemental-operator)

There are 3 different flavors of the SL Micro 6.2 image: bare-metal, kvm, rt (not checked here), as well as the base-image used by these flavors.

Testing elemental in a virtualized environment, we used the kvm flavor to deploy clusters, experiencing severe issues, like a missing iptables-package.

Turns out: While the base image contains a lot of elemental-related dependencies, the important stuff for rke2-clusters is installed in the bare-metal-image. This makes the bare-metal images the only usable flavor. The other flavors are based on the base-image and don't add the required dependencies defined in the bare-metal dockerfile.

I'd suggest adding the rke2-requirements of the bare-metal dockerfile to the base image, or change the bases of the other 2 flavors to use the bare-metal image instead.

[kkaempf]

The initial k8s target for Elemental was k3s. That's what the -kvm image was created for.

Adding a few dependencies is probably doable.
Re-basing on top of bare-metal should not be done, as this adds 300-500MB of (useless in a virtual environment) driver and firmware packages.

[kkaempf]

Btw, you're explicitly welcome to build your own images based on the provided "baseline" ones 😉

[max06]

Oh, I see. It looked like the kvm image provided the same functionality as the bare-metal one just minus the whole hardware stuff. Turned out the bare-metal image not only has iptables, but also ssh-server and other quality-of-life improvements. So we simply updated to the bare-metal version instead, issues gone.

You're doing a great job with these images, no need the re-invent the wheel. But I agree, we might have to start building our own images soon, when we finalized the network.

The last time I tried that (about a year ago) it didn't went very well. We needed frr-support in network-manager but somehow packages collided with your branding-elemental package. I think we ended up building images from scratch. Let's see, I'm gonna put this on my todo-list for next weekend... 😅

[kkaempf]

Here's the thinking behind the various image flavors: 😉

• base: focus on minimal size and minimal package selection (to run k3s) in a virtual environment. Base for all other images. Drop a single package and it becomes useless. Don't add any fluff.
• kvm: focus on minimal size and make it useful in a virtual environment - chose a minimal kernel, add qemu-guest-agent.
• baremetal: add drivers and firmware for bare metal deployment, add "usability" packages without considering final image size.
• rt: bare metal with a realtime kernel

while you can use these images as-is, users are explicitly encouraged to build their own images (just like you add/remove packages from a normal Linux installation).

[kkaempf]

frr-support in network-manager but somehow packages collided with your branding-elemental package.

Details please 🥺 (maybe in a separate issue ?)

[kkaempf]

Btw, feel free to reach out via the #elemental channel on the Rancher Users Slack.

[max06]

Hey @kkaempf, sorry about the delay - too many parties to attend.

Thanks for the clarification on the image flavors! Using the kvm flavor was the right way to go for us.

I think the iptables package should be moved from the baremetal image to the base image, since it seems to be required in rke2-clusters with calico and cilium as well. Our pure-kvm test-cluster pretty much didn't work without it.

We'll probably start building our own images, simply to have control about the package selection. Dependencies tend to break in the worst moment possible and building fixed versions in OBS is not the fastest experience 😅

I already found a nice improvement, that might fit here: You have some dependencies to elemental-toolkit and elemental-register in your build process. Your makefile depends on existing docker images, either locally or by specifying known urls. It's not very intuitive if you just started with elemental. Docker bake might be a fitting utility for this:

target "elemental-toolkit" {
  dockerfile = "Dockerfile"
  context = "https://github.com/rancher/elemental-toolkit.git"
  platforms = ["linux/amd64"]
}

target "elemental-os" {
  contexts = {
    toolkit = "target:elemental-toolkit"
  }
  platforms = ["linux/amd64"]
  tags = ["elemental-os:latest"]
}

Each context is available like in a multistage-dockerfile: COPY --from=toolkit /usr/bin/elemental /usr/bin/elemental

I've been looking for the ability to use multiple dockerfiles from different projects in my build process without copying their contents into my dockerfile for a while now. Had no idea bakefiles would actually just do that, and it works great!

Details please 🥺 (maybe in a separate issue ?)

I'd love to but it's been too long ago, even changed company in the meantime.
As much as I can remember: The rolling nature of tumbleweed conflicted with the snapshot saved in your images, requiring updates for most of the packages to be able to install additional packages. In this case, the version of NetworkManager-ovs offered by the tumbleweed repository wasn't compatible with the installed version of NM. In turn the systemd-presets-branding-Elemental package showed up in conflicts while trying to resolve the package changes. We moved to our own custom image by basically copying your bare-metal dockerfile and skipping packages we didn't need. Or in short: the older the base-image for a customization, the bigger the update could be. Not sure if your recommendation about customization in the docs is the best option...

Btw, feel free to reach out via the #elemental channel

Thanks a lot! I think I'm already in there. Will wave hands when questions arise 😄

[kkaempf]

Thanks for the update @max06, I've just opened #1753 about the iptables package.