Fix signature handling with additionalimagestore #166

gursewak1997 · 2025-11-27T22:43:44Z

Copy images to local storage without signatures before bootc install to avoid signature invalidation errors. Falls back to original behavior if copy fails.

Closes: #126

gemini-code-assist

Code Review

This pull request introduces a workaround to handle image signature invalidation errors by copying the image without signatures before installation. My review focuses on improving the robustness of the added shell script logic. I've suggested using trap for reliable temporary file cleanup and quoting variables to prevent potential shell injection or word-splitting issues.

crates/kit/src/to_disk.rs

cgwalters · 2025-12-01T21:55:08Z

crates/kit/src/to_disk.rs

+            cat > "${SIG_POLICY}" <<'EOF'
+{"default":[{"type":"insecureAcceptAnything"}],"transports":{"containers-storage":[{"type":"insecureAcceptAnything"}]}}
+EOF
+            if skopeo copy --signature-policy "${SIG_POLICY}" --remove-signatures \


It's not clear to me what the if is doing here. If skopeo fails due to being say out of disk space then we just...try to ignore that? That seems odd.

crates/kit/src/to_disk.rs

cgwalters

Offhand looks OK to me. Does it work?

cgwalters · 2025-12-03T19:24:39Z

crates/kit/src/to_disk.rs

+            
+            # Create permissive policy.json (use /var/tmp since it's mounted into podman container)
+            # Mount as directory to /etc/containers so podman creates the directory if it doesn't exist
+            POLICY_DIR=/var/tmp/bcvk-policy-dir


I know this is an ephemeral isolated system but it's probably still good to avoid hardcoded filenames in /tmp or /var/tmp. So we could use e.g. policydir=$(mktemp -d) or so.

It might be a nicer cleanup to have this policy JSON string as a literal file in the Rust source that we include via include_str! and then template into the shell script.

cgwalters · 2025-12-03T19:24:59Z

crates/kit/src/to_disk.rs

                --env=STORAGE_OPTS \
                {INSTALL_LOG} \
-                {SOURCE_IMGREF} \
+                "${SOURCE_REF}" \


This change looks unrelated?

gursewak1997 · 2025-12-03T19:27:15Z

Offhand looks OK to me. Does it work?

Some minor issues rn. Working on some changes to make the integration tests pass.

crates/kit/src/to_disk.rs

cgwalters

One nit, but looks sane to me; the key thing is verifying this works at least manually.

cgwalters · 2025-12-05T18:55:07Z

crates/kit/src/to_disk.rs

            # Execute bootc installation, having the outer podman pull from
            # the virtiofs store on the host, as well as the inner bootc.
            # Mount /var/tmp into inner container to avoid cross-device link errors (issue #125)
-            export STORAGE_OPTS=additionalimagestore=${AIS}


Why did this line move? See above

crates/kit/src/to_disk.rs

gursewak1997 · 2025-12-05T19:44:35Z

One nit, but looks sane to me; the key thing is verifying this works at least manually.

Did manually test it and the integration test test_to_disk_for_image_quay_io_almalinuxorg_almalinux_bootc_10_0 passed, which exercises the same code path.

cgwalters · 2025-12-05T21:50:48Z

I edited this issue just now to have Closes: #126 but let's try to be sure we're doing that kind of thing in general in the future

cgwalters · 2025-12-05T21:52:19Z

Hmmm I get the same error trying this out with the reproducer from #126

cgwalters · 2025-12-09T15:05:29Z

@gursewak1997 what's the status of this one? Did it work for you?

gursewak1997 · 2025-12-09T19:39:23Z

@gursewak1997 what's the status of this one? Did it work for you?

this fails since podman validates signatures when starting the container from the signed image, before our copy script runs.

cgwalters · 2025-12-10T17:11:16Z

this fails since podman validates signatures when starting the container from the signed image, before our copy script runs.

Hmm that seems odd since we should have overridden the signature requirements before podman starts.

cgwalters · 2025-12-10T17:12:25Z

Ahh no I see. I think we just need to change from writing policy.json to a tempdir to write it to the VM's /etc/containers/policy.json and then bind mount from there into the container.

cgwalters · 2025-12-12T18:31:27Z

crates/kit/src/to_disk.rs

+EOF
+
+            # Copy image without signatures to avoid "Would invalidate signatures" error
+            skopeo copy --remove-signatures {SOURCE_IMGREF} containers-storage:{SOURCE_IMAGE}


But this is still putting us back in a world of doing a deep copy of the image, which I'd really like to avoid. Generating disk images is already too slow; gigabytes add up fast. We're doing this copying in multiple places (bootc-dev/bootc#1601 is also doing that).

This relates to containers/container-libs#144 which at least should have a comment here.

It looks to me like you dropped the bind mount of the policy into the podman run below - why is that? I thought that was an important part of the fix here because again it's also bootc install inside that runs skopeo which will also fetch/access the image.

Updating now:

You're right that this is important because bootc install internally runs skopeo which also needs to access the image. The bind mount ensures those internal operations work correctly.

Added comments referencing reflink by default (if possible) when copying between containers-storage: instances containers/container-libs#144 and lib: Add experimental unified storage support for install bootc#1601 to document this is a known workaround.

I tested the policy.json-only approach (without the skopeo copy) but it still fails with "Would invalidate signatures". Unfortunately the deep copy with --remove-signatures appears necessary for now.

Re: containers/container-libs#144 - Fixing that issue would make the copy much faster (via reflinks) and use less disk space, but we'd still need to copy because we're removing signatures.
The real fix would need to be in containers/image or bootc to handle signature validation differently when layer representation changes. Until then, afaik we're stuck with the copy, but at least it happens at the VM level using the additionalimagestore mount from the host rather than pulling over the network.
Open to other ideas though:)

So bootc-image-builder did this unconditionally: osbuild/bootc-image-builder@22de964

But it was already (unnecessarily, IMO incorrectly) copying the whole container image anyways, so it didn't matter. (A lot of related discussion in #73 )

I dunno...I still think there should be some way (arguably, it might be something in bootc or skopeo like --ignore-signature-policy or so) to skip this requirement.

But I think even that's not the real fix...the real fix is we should be able to carry across the signature (which is actually containers/image#2590 )

Anyways...there is another option in the short term, which is to force this onto the user to decide via like bcvk to-disk --remove-signatures or so (and we'd need that in libvirt run too of course).

Maybe we could hackily parse the error message and look for that string and actually fall back to the copy only if we get that error...

Looking short term for now: I like the second option; parse error and fallback automatically. Try without copy first, fallback to copy if we see "Would invalidate signatures":

First attempt: direct use (fast, preserves signatures if possible)
podman run ... {SOURCE_IMGREF} bootc install to-disk ...
If that fails with "Would invalidate signatures":

skopeo copy --remove-signatures {SOURCE_IMGREF} containers-storage:{SOURCE_IMAGE} podman run ... containers-storage:{SOURCE_IMAGE} bootc install to-disk ...

Also, when containers/image#2590 is fixed, we automatically benefit (no more fallback needed)

Try bootc install first using the image from additionalimagestore. If it fails with "Would invalidate signatures", automatically retry with skopeo copy --remove-signatures as a fallback. This avoids the deep copy when possible (e.g., for unsigned images or when containers/image#2590 is fixed), while still working with signed images like RHEL that currently fail. Fixes: #126 Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: gursewak1997 <gursmangat@gmail.com>

cgwalters · 2025-12-15T20:05:46Z

crates/kit/src/to_disk.rs

+                skopeo copy --remove-signatures {SOURCE_IMGREF} containers-storage:{SOURCE_IMAGE}
+
+                # Retry bootc install with the unsigned local copy
+                podman run --rm -i ${tty} --privileged --pid=host --net=none -v /sys:/sys:ro \


One thing I think would help here is to deduplicate this into a shared variable

gursewak1997 force-pushed the fix/to-disk-signatures branch from dadf28e to a8ed770 Compare November 27, 2025 22:44

gemini-code-assist bot reviewed Nov 27, 2025

View reviewed changes

crates/kit/src/to_disk.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/to-disk-signatures branch from a8ed770 to a710bde Compare November 27, 2025 22:48

gursewak1997 linked an issue Nov 28, 2025 that may be closed by this pull request

to-disk: Handle signatures #126

Closed

cgwalters reviewed Dec 1, 2025

View reviewed changes

gursewak1997 force-pushed the fix/to-disk-signatures branch from a710bde to 0b83293 Compare December 1, 2025 23:04

cgwalters reviewed Dec 2, 2025

View reviewed changes

crates/kit/src/to_disk.rs Outdated Show resolved Hide resolved

crates/kit/src/to_disk.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/to-disk-signatures branch 3 times, most recently from e25b44d to 94a36f6 Compare December 3, 2025 00:03

cgwalters reviewed Dec 3, 2025

View reviewed changes

gursewak1997 force-pushed the fix/to-disk-signatures branch from 94a36f6 to ea1cb15 Compare December 3, 2025 20:24

cgwalters reviewed Dec 3, 2025

View reviewed changes

crates/kit/src/to_disk.rs Outdated Show resolved Hide resolved

gursewak1997 force-pushed the fix/to-disk-signatures branch from ea1cb15 to e68793b Compare December 5, 2025 18:38

cgwalters reviewed Dec 5, 2025

View reviewed changes

gursewak1997 force-pushed the fix/to-disk-signatures branch from e68793b to c1d7037 Compare December 5, 2025 19:37

gursewak1997 force-pushed the fix/to-disk-signatures branch 2 times, most recently from 9592974 to 914df59 Compare December 6, 2025 06:54

gursewak1997 force-pushed the fix/to-disk-signatures branch from 914df59 to edb88ea Compare December 12, 2025 00:44

gursewak1997 requested a review from cgwalters December 12, 2025 17:57

cgwalters requested changes Dec 12, 2025

View reviewed changes

gursewak1997 force-pushed the fix/to-disk-signatures branch from edb88ea to bdfdc98 Compare December 12, 2025 20:06

gursewak1997 force-pushed the fix/to-disk-signatures branch 3 times, most recently from a6bb8a7 to b7f6ded Compare December 12, 2025 21:49

gursewak1997 force-pushed the fix/to-disk-signatures branch from b7f6ded to 5cb1a1b Compare December 12, 2025 23:50

cgwalters approved these changes Dec 15, 2025

View reviewed changes

cgwalters merged commit d2ed636 into main Dec 15, 2025
7 checks passed

gursewak1997 deleted the fix/to-disk-signatures branch December 15, 2025 20:37

Fix signature handling with additionalimagestore #166

Fix signature handling with additionalimagestore #166

Uh oh!

Conversation

gursewak1997 commented Nov 27, 2025 • edited by cgwalters Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

cgwalters Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

gursewak1997 commented Dec 3, 2025

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gursewak1997 commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Dec 5, 2025

Uh oh!

cgwalters commented Dec 5, 2025

Uh oh!

cgwalters commented Dec 9, 2025

Uh oh!

gursewak1997 commented Dec 9, 2025

Uh oh!

cgwalters commented Dec 10, 2025

Uh oh!

cgwalters commented Dec 10, 2025

Uh oh!

cgwalters Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

gursewak1997 Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

gursewak1997 Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gursewak1997 commented Nov 27, 2025 •

edited by cgwalters

Loading

gursewak1997 commented Dec 5, 2025 •

edited

Loading

gursewak1997 Dec 12, 2025 •

edited

Loading

gursewak1997 Dec 12, 2025 •

edited

Loading