Feature: Add a persistent backend to the cloud-init server which uses duckdb via quack/quack

[alexlovelltroy]

This pull request introduces a new storage backend, quackstore, and includes various updates to support this addition. The most significant changes involve modifying the main application logic to handle the new backend, adding the necessary dependencies, and implementing and testing the quackstore functionality.

Main Application Updates:

• Added quackstore import and new flags for selecting the storage backend and database path in cmd/cloud-init-server/main.go. [1] [2] [3]
• Updated the main function to initialize the datastore based on the selected backend (memstore or quackstore).

New quackstore Implementation:

• Implemented quackstore package with methods for managing groups, instances, and cluster defaults, including initialization and schema setup.

Testing Enhancements:

• Added a test for memstore to ensure it adheres to the standard test suite.
• Added tests for quackstore to ensure proper functionality and integration.

Model Serialization:

• Enhanced CloudConfigFile struct to implement custom JSON marshaling and unmarshaling, handling different encoding types (base64 and plain).

[Copilot]

Pull Request Overview

This PR introduces support for a new persistent storage backend (quackstore) using duckdb via quack/quack while maintaining support for the in-memory memstore. It updates the main application flags and initialization logic, adds new tests for quackstore, and refactors model JSON marshalling functions.

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/cistore/testing/store_test_suite.go	New tests for verifying store implementations.
pkg/cistore/models_test.go	Tests for CloudConfigFile custom JSON marshaling/unmarshaling.
pkg/cistore/models.go	Refactored JSON logic for CloudConfigFile serialization.
internal/quackstore/quackstore_test.go	Tests for quackstore including DB initialization and cleanup.
internal/quackstore/quackstore.go	Implementation of quackstore; schema initialization and API methods.
internal/memstore/ciMemStore_test.go	Standard in-memory store tests.
cmd/cloud-init-server/main.go	Updated flag parsing and backend switching logic for storage backend.

Files not reviewed (1)

• go.mod: Language not supported

[synackd]

Any progress on this?

[synackd]

Thanks @alexlovelltroy. Do you have instructions for testing this?

[synackd]

Also:

• Looks like there have been some new versions since the last merge from main that we'll probably want to incorporate.
• Looks like DCO check is failing.

[alexlovelltroy]

@synackd This PR adds two flags to control the behavior of the storage backend.

      --db-path string            Path to the database file for quackstore backend (default "cloud-init.db")
      --storage-backend string    Storage backend to use (mem or quack) (default "mem")

In addition to running the unit tests in the internal/quackstore directory, you can start the cloud-init server with these flags and use it normally. To test the durable storage, feel free to kill -9 cloud-init-server and then restart. It should retain any information that was loaded previously.

In a containerized deployment, the files should be stored in a volume that is remounted correctly when restarting the container. It is easier to test running manually though.

[synackd]

I'm currently testing using a modification of the cloud-init quadlet in the release repo, using a locally-built cloud-init container using buildah bud:

/etc/containers/systemd/cloud-init-server.container

[Unit]
Description=The cloud-init-server container
Wants=smd.service
After=smd.service opaal.service
PartOf=openchami.target

[Container]
ContainerName=cloud-init-server
HostName=cloud-init
Image=ghcr.io/openchami/cloud-init:test

Volume=cloud-init:/cloud-init

# Environment Variables
EnvironmentFile=/etc/openchami/configs/openchami.env

# Networks for the Container to use
Network=openchami-internal.network

# Proxy settings
PodmanArgs=--http-proxy=false

[Service]
Restart=always

I've also created the corresponding volume:

/etc/containers/systemd/cloud-init.volume

[Unit]
Description=cloud-init Volume

[Volume]
VolumeName=cloud-init

I've modified /etc/openchami/configs/openchami.env to have:

STORAGE_BACKEND=quack
DB_PATH=/cloud-init/cloud-init.db

After reloading systemd and (re)starting the container, the following fatal error appears in the journal:

cloud-init-server[262813]: /usr/local/bin/cloud-init-server: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

Adding the libstdc++-6 package to the Dockerfile doesn't seem to fix it. I think this might be due to the CGO enablement in the last commit.

[synackd]

Building via goreleaser yields the same thing.

[alexlovelltroy]

Cross-platform builds with CGO are easier on ubuntu and should require a glibc container rather than a musl one. I've updated the container and added a build-in-container script that should replicate the way github will do the builds.

[synackd]

With the most recent commit, I get the following error in Goreleaser when running goreleaser release --clean --snapshot --skip publish,archive:

  ⨯ release failed after 15s                        
    error=
    │ docker build failed: failed to build ghcr.io/openchami/cloud-init:v1.2.3-amd64: exit status 1: STEP 1/10: FROM registry.access.redhat.com/ubi9/ubi-minimal
    │ Trying to pull registry.access.redhat.com/ubi9/ubi-minimal:latest...
    │ Getting image source signatures
    │ Checking if image destination supports signatures
    │ Copying blob sha256:e7dc8c72147da3785cb494f592a4596697015bf12fb34fe8b2bf842d6105b702
    │ Copying config sha256:04e3acee198ee12f9f16a6f99ff124d7cc0d7e73971f2386e0cf353717cac13d
    │ Writing manifest to image destination
    │ Storing signatures
    │ STEP 2/10: RUN microdnf install -y         libstdc++         libgcc         curl         iproute         iputils         wireguard-tools         tini     && microdnf clean all
    │ (microdnf:2): librhsm-WARNING **: 23:17:41.323: Found 0 entitlement certificates
    │ (microdnf:2): librhsm-WARNING **: 23:17:41.324: Found 0 entitlement certificates
    │ Downloading metadata...
    │ Downloading metadata...
    │ Downloading metadata...
    │ error: No package matches 'wireguard-tools'
    │ Error: building at STEP "RUN microdnf install -y         libstdc++         libgcc         curl         iproute         iputils         wireguard-tools         tini     && microdnf clean all": while running runtime: exit status 1
    │ Learn more at https://goreleaser.com/errors/docker-build

[alexlovelltroy]

I'll work on that in the morning. Can you test the standalone binary in the meantime?

[alexlovelltroy]

Build automation is noble yak shaving... The yak never gets any smaller. At least this time I think the builds are happy.

[synackd]

I finally got the container to build with the script after applying the suggested edits.

However, when starting the container, I see:

cloud-init-server[2766863]: /usr/local/bin/cloud-init-server: /lib64/libm.so.6: version `GLIBC_2.38' not found (required by /usr/local/bin/cloud-init-server)
cloud-init-server[2766863]: /usr/local/bin/cloud-init-server: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /usr/local/bin/cloud-init-server)

Run running the binary itself via

./cloud-init-server --storage-backend quack --db-path ./cloud-init.db

it panics:

No JWKS URL provided; secure route will be disabled
5:32PM ERR Failed to get SMD data error="Get \"http://smd:27779/hsm/v2/Inventory/EthernetInterfaces/\": dial tcp: lookup smd: no such host"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xab17f4]

goroutine 1 [running]:
github.com/go-chi/chi/v5.chain({0xc00038a0c8, 0x1, 0x2779680?}, {0x2925720?, 0x281f2a0?})
        /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.1/chain.go:43 +0x34
github.com/go-chi/chi/v5.Middlewares.Handler(...)
        /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.1/chain.go:13
github.com/go-chi/chi/v5.(*Mux).handle(0xc0003842a0, 0x8, {0x27ca46e, 0xa}, {0x2925720, 0x281f2a0})
        /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.1/mux.go:436 +0x1ae
github.com/go-chi/chi/v5.(*Mux).Get(0xc000384240?, {0x27ca46e?, 0x1?}, 0x735e79?)
        /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.1/mux.go:162 +0x2d
main.initCiClientRouter({0x293f9c0, 0xc000384240}, 0xc000051c40, 0x0)
        /opt/shared/cloud-init/cmd/cloud-init-server/main.go:265 +0xeb
main.startServer()
        /opt/shared/cloud-init/cmd/cloud-init-server/main.go:228 +0xbeb
main.main.func1(0xc000058400?, {0x27c4c6e?, 0x4?, 0x27c4b4a?})
        /opt/shared/cloud-init/cmd/cloud-init-server/main.go:65 +0xf
github.com/spf13/cobra.(*Command).execute(0xc0001d5b08, {0xc000034060, 0x4, 0x4})
        /root/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1015 +0xa94
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001d5b08)
        /root/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(...)
        /root/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071
main.main()
        /opt/shared/cloud-init/cmd/cloud-init-server/main.go:75 +0x6f

Though this doesn't seem to be related to this pull request. Applying the following fix:

diff --git a/cmd/cloud-init-server/main.go b/cmd/cloud-init-server/main.go
index 9b5113a..9cb98f6 100644
--- a/cmd/cloud-init-server/main.go
+++ b/cmd/cloud-init-server/main.go
@@ -262,10 +262,17 @@ func initCiClientRouter(router chi.Router, handler *CiHandler, wgInterfaceManage
        // Add cloud-init endpoints to router
        router.Get("/openapi.json", DocsHandler)
        router.Get("/version", VersionHandler)
-       router.With(wireGuardMiddleware).Get("/user-data", UserDataHandler)
-       router.With(wireGuardMiddleware).Get("/meta-data", MetaDataHandler(handler.sm, handler.store))
-       router.With(wireGuardMiddleware).Get("/vendor-data", VendorDataHandler(handler.sm, handler.store))
-       router.With(wireGuardMiddleware).Get("/{group}.yaml", GroupUserDataHandler(handler.sm, handler.store))
+       if wireGuardMiddleware != nil {
+               router.With(wireGuardMiddleware).Get("/user-data", UserDataHandler)
+               router.With(wireGuardMiddleware).Get("/meta-data", MetaDataHandler(handler.sm, handler.store))
+               router.With(wireGuardMiddleware).Get("/vendor-data", VendorDataHandler(handler.sm, handler.store))
+               router.With(wireGuardMiddleware).Get("/{group}.yaml", GroupUserDataHandler(handler.sm, handler.store))
+       } else {
+               router.Get("/user-data", UserDataHandler)
+               router.Get("/meta-data", MetaDataHandler(handler.sm, handler.store))
+               router.Get("/vendor-data", VendorDataHandler(handler.sm, handler.store))
+               router.Get("/{group}.yaml", GroupUserDataHandler(handler.sm, handler.store))
+       }
        router.Post("/phone-home/{id}", PhoneHomeHandler(wgInterfaceManager, handler.sm))
        router.Post("/wg-init", wgtunnel.AddClientHandler(wgInterfaceManager, handler.sm))
 }

and I was finally able to test! :)

I did some very basic tests and I was able to persistently store cluster-defaults and group data.

[synackd]

The container successfully builds now with e626748.

[synackd]

I wonder if the issue is that we are building the binary using an Ubuntu container but it is being run in a Rocky Linux 9 container.

[alexlovelltroy]

Finally got the DCO to pass. @synackd can you recheck?

[synackd]

Glibc version errors are still occurring in the container. This is because ubuntu:24.04 is being used to build the cloud-init binary, which is then copied into a rockylinux:9 container to be run. Since CGO_ENABLED=1, the binary depends on a specific Glibc version, one that differs between the two containers.

I think the solution is to build in the same container which cloud-init will run, rockylinux:9. I have gotten a successful build working on my branch here: synackd/container-build-fixes. It adds a separate .goreleaser-buildincontainer.yaml file that sets appropriate linker variables for ARM, as well as a separate build-in-container-r9.sh build script. These were added as separate files so that the differences can be seen with the existing ones.

With docker:

./build-in-container-r9.sh

With Podman (what I've tested with, Podman socket needs to be enable):

CONTAINER_CMD=podman CONTAINER_SOCK=/run/podman/podman.sock ./build-in-container-r9.sh

[synackd]

Now that I've gotten the container running, I've tried setting:

STORAGE_BACKEND=quack
DB_PATH=/cloud-init/cloud-init.db
DEBUG=true

and am using the following quadlet with an existing OpenCHAMI installation (the cloud-init volume is a simple Podman volume with no custom config):

[Unit]
Description=The cloud-init-server container
Wants=smd.service
After=smd.service opaal.service
PartOf=openchami.target

[Container]
ContainerName=cloud-init-server
HostName=cloud-init
Image=ghcr.io/openchami/cloud-init:v1.2.3-amd64

Volume=cloud-init:/cloud-init:rw,Z

# Environment Variables
EnvironmentFile=/etc/openchami/configs/openchami.env

# Networks for the Container to use
Network=openchami-internal.network

# Proxy settings
PodmanArgs=--http-proxy=false

[Service]
Restart=always

I can see the DB path and enablement get picked up, but DuckDB is attempting to create a directory in the root directory at /.duckdb, which errs due to a permissions error:

cloud-init-server[426685]: {"level":"debug","listen":":27777","token-url":"http://opaal:3333/token","smd-url":"http://smd:27779","jwks-url":"http://opaal:3333/keys","access-token":"","cluster-name":"","region":"","az":"","cloud-provider":"","base-url":"","cacert":"","insecure":false,"impersonation":true,"wireguard-server":"","wireguard-only":false,"debug":true,"storage-backend":"quack","db-path":"/cloud-init/cloud-init.db","time":"2025-08-15T23:17:38Z","message":"Resolved configuration"}
cloud-init-server[426685]: 11:17PM ERR Failed to load DuckDB extensions error="IO Error: Failed to create directory \"/.duckdb/\": Permission denied"

This seems to be occurring in the loadExtensions() function here. From what I can tell, the path set in DB_PATH gets propagated to the DB creation, so I'm not sure where this is occurring or how it's choosing where to write.

[synackd]

Glibc version errors are still occurring in the container. This is because ubuntu:24.04 is being used to build the cloud-init binary, which is then copied into a rockylinux:9 container to be run. Since CGO_ENABLED=1, the binary depends on a specific Glibc version, one that differs between the two containers.

I think the solution is to build in the same container which cloud-init will run, rockylinux:9. I have gotten a successful build working on my branch here: synackd/container-build-fixes. It adds a separate .goreleaser-buildincontainer.yaml file that sets appropriate linker variables for ARM, as well as a separate build-in-container-r9.sh build script. These were added as separate files so that the differences can be seen with the existing ones.

With docker:

./build-in-container-r9.sh

With Podman (what I've tested with, Podman socket needs to be enable):

CONTAINER_CMD=podman CONTAINER_SOCK=/run/podman/podman.sock ./build-in-container-r9.sh

Upon further discussion, it might be better to change the container that cloud-init runs in from rockylinux:9 to ubuntu:24.04 since GitHub actions uses an Ubuntu runner and it's easier to cross compile in Ubuntu than it is in Rocky Linux.

[alexlovelltroy]

I tried several different things, but switching to an Ubuntu base container makes the CGO stuff work better with the fewest complications.

[synackd]

I was attempting to test with ochami and was getting a bunch of 404s. Looks like the API got modified in an undocumented way which breaks functionality. See my review comment.

[synackd]

I tried several different things, but switching to an Ubuntu base container makes the CGO stuff work better with the fewest complications.

Other than my review above, the container build works with the new ubuntu changes. I still see the DuckDB extension permission error when running the container, but cloud-init runs in the container now.

[synackd]

There are a lot of places where errors are being ignored, many of which I don't quite understand if there's a legitimate reason or not. If there is a legitimate reason, they should be commented (I did see some comments already) since ignoring errors in production code is generally not a good practice since it makes it hard to troubleshoot where problems are arising.

Besides that, there were a few idiomatic suggestions I added.

This was purely a code review and I still need to test it.

[synackd]

Other than the DuckDB plugins error still occurring, everything else appears to work.

[synackd]

Here's the error I'm still getting, which occurs when cloud-init starts:

cloud-init-server[3844715]: 9:26AM ERR Failed to load DuckDB extensions error="IO Error: Can't find the home directory at '/nonexistent'\nSpecify a home directory using the SET home_directory='/path/to/dir' option." home=/nonexistent 

[synackd]

For some of the errors we are ignoring, I'd be more comfortable if we logged them even if we don't actually handle them. That way any unexpected behavior presents enough information for the administrator to troubleshoot where an issue originates. I pushed a commit that adds this.

Besides that, now I'm seeing:

cloud-init-server[178104]: 12:30AM ERR Failed to load DuckDB extensions error="IO Error: Failed to download extension \"json\" at URL \"http://extensions.duckdb.org/v1.1.3/linux_amd64/json.duckdb_extension.gz\"\nExtension \"json\" is an existing extension.\n (ERROR Could not establish connection)" home=/tmp/duckdbhome

when cloud-init starts.

[alexlovelltroy]

@synackd Can you confirm that you don't get that error anymore? I'm building the container differently now so we can preload all plugins which is also safer.

[synackd]

This is what I see now:

cloud-init-server[1952550]: 2025/09/18 20:29:38 failed to create modcache index dir: mkdir /nonexistent: permission denied
...
cloud-init-server[1952550]: 8:29PM ERR Failed to load DuckDB extensions error="IO Error: Failed to download extension \"json\" at URL \"http://extensions.duckdb.org/v1.1.3/linux_amd64/json.duckdb_extension.gz\"\nExtension \"json\" is an existing extension.\n (ERROR Could not establish connection)" home=/opt/duckdbhome

[alexlovelltroy]

This is with the goreleaser-built docker container?

[synackd]

Yep, built with:

CONTAINER_CMD=podman CONTAINER_SOCK=/run/podman/podman.sock ./build-in-container.sh

[alexlovelltroy]

I added more error handling to the quack library and refactored the way extensions are loaded.

[synackd]

I no longer see the error! Are there any more impending changes that need to be made?

[alexlovelltroy]

Not that I'm aware of. Once you approve this PR, we can merge it.

[synackd]

I've pushed a commit updating the documentation of the removal of /cloud-init. Let me know your thoughts on this. I think this is the last change needed before merging.

[synackd]

Mounting things at / instead of /cloud-init is a welcome change, but it will be a breaking change to users of the release RPM unless

http-request replace-path ^/cloud-init(/.*) \1

is added to the cloud-init backend in the HAproxy config.

We can certainly make this change when incorporating this PR into the release repo, but I think it's worth pointing out here for posterity.

[synackd]

Reflecting some more, wouldn't this technically be an API change? Not sure we want to do that in this PR. Thoughts?

[synackd]

Missed a few things, but commit is up-to-date now.

[alexlovelltroy]

this PR requires an update to haproxy before we can make it a release.

[synackd]

I think this is ready. Making a note here that the endpoints are mounted at / with this PR instead of /cloud-init, so we'll need to note that when making a release.