Cache builds of ZFS

[bsherman]

Need to build ZFS for both debian-main and debian-backports, cache in an appropriate mechanism.

Include adding signing keys to secrets as appropriate

[bketelsen]

— Automated by Yeti —

Implementation Plan

Now I have a full picture of the repo. Here's the implementation plan:

---

Implementation Plan: Cache Builds of ZFS (Issue #2)

Context

Cayo is a Debian bootc server image. The build_files/build script already has a TODO for ZFS & sanoid/syncoid (line 133). ZFS on Debian requires DKMS kernel module compilation, which is slow and needs build tools. The issue asks to pre-build ZFS modules for both the debian-main and debian-backports kernels, cache them as a reusable artifact, and set up signing keys.

Architecture

The approach: create a separate OCI image (cayo-zfs-cache) containing pre-compiled ZFS kernel modules and .deb packages. The main Containerfile mounts this cache image during build to install ZFS without compiling from source. A separate GitHub Actions workflow builds and publishes the cache image on a schedule and when triggered.

---

Files to Create

1. Containerfile.zfs-cache

A multi-stage Containerfile that:

• Starts FROM ghcr.io/frostyard/debian-bootc-core:latest
• Enables contrib component (ZFS lives in contrib on Debian)
• Enables backports with contrib
• Installs build dependencies: linux-headers-*, dkms, build-essential, zfs-dkms (from debian-main)
• Copies the compiled .ko modules and any generated .deb files to a scratch stage
• Repeats for backports kernel (or uses a build arg to select kernel variant)
• Final image is a minimal scratch/busybox image containing only the pre-built module .deb files organized by kernel version

Structure inside the cache image:

/cache/
  main/          # modules built against debian-main kernel
    *.deb
  backports/     # modules built against debian-backports kernel
    *.deb

2. .github/workflows/build-zfs-cache.yml

New workflow that:

• Triggers: workflow_dispatch, schedule (weekly), and when kernel-related packages update
• Matrix: builds for both main and backports kernel variants
• Steps:
  1. Checkout repo
  2. Set up container storage (same BTRFS pattern as build.yml)
  3. Build Containerfile.zfs-cache with build args for kernel variant
  4. Login to GHCR
  5. Push to ghcr.io/frostyard/cayo-zfs-cache:{main,backports,latest}
  6. Sign with cosign using secrets.SIGNING_SECRET
• Permissions: contents: read, packages: write, id-token: write

3. build_files/install-zfs

A helper script (called from build_files/build) that:

• Detects the running kernel version
• Determines whether it's a main or backports kernel
• Installs the matching pre-built ZFS .deb packages from the mounted cache
• Installs ZFS userspace tools (zfsutils-linux) from apt
• Installs sanoid for snapshot management (syncoid for replication)
• Loads/verifies the ZFS module can be loaded (or at minimum verifies the .ko files exist for the target kernel)

---

Files to Modify

4. Containerfile

Add a new stage pulling the ZFS cache image and mount it during build:

FROM ghcr.io/frostyard/cayo-zfs-cache:latest AS zfs-cache

# ... existing ctx stage ...

# In the main RUN, add mount:
RUN --mount=type=bind,from=ctx,source=/,target=/ctx \
    --mount=type=bind,from=zfs-cache,source=/cache,target=/zfs-cache \
    --mount=type=cache,dst=/var/cache \
    ...

This follows the existing pattern of FROM scratch AS ctx with bind mounts.

5. build_files/build

• Add contrib to both the main sources and backports sources (ZFS packages are in contrib):

  Components: main contrib non-free non-free-firmware
  

• After the existing package installs, call /ctx/install-zfs (or inline the ZFS userspace package installation)
• Install: zfsutils-linux, zfs-zed (ZFS event daemon), sanoid
• Remove the # zfs & sanoid/syncoid TODO comment
• The pre-built kernel modules come from the cache mount, userspace tools from apt

6. .github/workflows/build.yml

• Uncomment the cosign signing block (lines 146-163) and make it active
• Update it to use the correct secret name (SIGNING_SECRET)
• Optionally add a needs: dependency on the ZFS cache workflow or at minimum document that the cache image must exist

---

GitHub Secrets to Configure (Manual Steps)

1. SIGNING_SECRET: Generate a cosign key pair (cosign generate-key-pair) and store the private key as a GitHub repository secret. The public key (cosign.pub) already exists in the repo root — verify it matches or update it.
2. Document in the workflow or README that these secrets must be configured.

---

Suggested Order of Implementation

1. Containerfile.zfs-cache — Define the ZFS module build container
2. .github/workflows/build-zfs-cache.yml — Workflow to build and push the cache image
3. build_files/install-zfs — Helper script to install from the cache
4. build_files/build — Add contrib to sources, add ZFS userspace packages, call install-zfs
5. Containerfile — Add the zfs-cache stage and mount
6. .github/workflows/build.yml — Uncomment cosign signing

---

Risks and Edge Cases

• Kernel version mismatch: The ZFS cache must be rebuilt whenever the kernel is updated in either main or backports. The cache workflow should run on a schedule (weekly) and be triggerable manually. The main build should fail clearly if the cached modules don't match the installed kernel rather than silently producing a broken image.
• DKMS build failures: ZFS DKMS can fail if kernel headers are missing or if there's a ZFS/kernel version incompatibility. The cache workflow should test module loading or at least verify .ko files were produced.
• Backports availability: ZFS packages may not always be in backports for the current Debian release. The build should handle this gracefully (e.g., fall back to main).
• sanoid availability: sanoid may not be in the standard Debian repos and might need to be installed from backports or a third-party source. Verify availability; if not packaged, it can be installed from GitHub releases or deferred to a follow-up.
• Cache staleness: If the cache image is stale and the main build uses a newer kernel, the ZFS modules won't load. Consider adding a version check in install-zfs that errors out with a clear message.
• Signing key rotation: Document how to rotate the cosign key pair. The cosign.pub in the repo must be updated if the key is regenerated.
• Build time on CI: ZFS DKMS compilation for two kernel variants will take 10-20+ minutes. This is acceptable since it runs separately from the main image build.
• Image size: ZFS kernel modules and userspace tools will add ~50-100MB to the final image. This is expected for a server image.
• Repo is deprecated: The README says this repo is deprecated in favor of frostyard/snosi. Confirm with maintainers that this work should still land here, or whether it should target snosi instead.