[RFC]: Proposal for Expanding Ignition with Cloud-Specific Configuration

[peytonr18]

Proposal for Expanding Ignition with Cloud-Specific Configuration

The majority of Azure VMs currently use cloud-init for VM provisioning. WALinuxAgent provisioning is being deprecated in favor of azure-init, and we are evaluating how best to support Flatcar and Fedora CoreOS. Upcoming changes to Azure Confidential VMs will encrypt custom data and sign user data, which is expected to break Ignition and prevent its use in CVM scenarios. Additionally, enabling Azure’s Metadata Security Protocol (MSP) requires Ignition to change how it fetches user data from IMDS due to the azure-proxy-agent requirement.

Azure-init can replace WALinuxAgent for tasks such as admin user creation, SSH key and password setup, and sshd configuration, but it does not address Ignition’s needs for accessing custom and user data.

This presents an opportunity to make Ignition a first-class provisioning agent on Azure, more closely aligned with cloud-init’s Azure datasource. Ignition’s ignition-fetch.service already overlaps with cloud-init’s local phase by mounting provisioning media and fetching instance metadata. The proposed --generate-cloud-config approach (see #2185) shows hows how cloud-specific behavior can be added by reusing Ignition’s existing configuration APIs as a clear, stable contract - similar to how Azure configuration is translated into cloud-init config. Afterburn does not address these gaps and is not well-suited as the contract surface for this functionality.

We can commit to bringing Ignition to parity with cloud-init on Azure and maintaining Azure-specific features, but this would expand Ignition’s scope. Community feedback is needed on whether this is acceptable. The fallback is to replace WALinuxAgent PA with azure-init and address Ignition’s limitations separately.

[travier]

If I understand correctly the description from #2185, this would change the behavior of Ignition to generate a config from metadata that it would fetch.

So how would this work if I provide an Ignition configuration to the node? Would the generated config be merged with the config I'm providing? How do we decide if the cloud metadata or the Ignition config would win? Some fields are not "overridable" in that I can not remove SSH keys in a merged config that have been added in another.

[cjp256]

If I understand correctly the description from #2185, this would change the behavior of Ignition to generate a config from metadata that it would fetch.

Yes, but it was not (intended) to conflict with user-provided ignition configuration. We missed that detail.

So how would this work if I provide an Ignition configuration to the node? Would the generated config be merged with the config I'm providing? How do we decide if the cloud metadata or the Ignition config would win? Some fields are not "overridable" in that I can not remove SSH keys in a merged config that have been added in another.

It appears merging is required. What do you think about this proposal...?

Proposal: Cloud-specific Ignition Extensions

Overview

Introduce Azure-specific Ignition rendering knobs under:

ignition.extensions.azure

These knobs control whether Azure emits and owns specific configuration artifacts
(e.g., users, drop-ins, mounts).

This metadata is consumed by ignition-fetch to control extension behavior.

---

Configuration

{
  "ignition": {
    "version": "3.6.0",
    "extensions": {
      "azure": {
        "mntResourceEnabled": true,
        "sshdDropInEnabled": true,
        "sudoersDropInEnabled": true,
        "userEnabled": true
      }
    }
  }
}

---

Example Azure-managed Artifacts

Knob	Artifact (path)
userEnabled	Azure-managed admin user (e.g., azureuser)
sudoersDropInEnabled	/etc/sudoers.d/azure-cloud-sudoers.conf
sshdDropInEnabled	/etc/ssh/sshd_config.d/50-azure-cloud-sshd.conf
mntResourceEnabled	mnt-resource.mount for mounting Azure's SCSI resource disk to /mnt/resource

All cloud-owned files are prefixed azure-cloud-* for clarity.

---

Conflict Handling (Strict, Fail‑fast)

If an Azure-managed artifact is enabled and the user’s Ignition config also defines the same user or drop-in path:

• Rendering fails, and the system service exits with failure
  OR
• We fail (exit with error) but still render a config that persists the user-provided ignition config to allow the system to come up mostly configured as expected. On Azure we would later report this as a failure to the platform to make it obvious to VM owner.

This ensures:

• No silent overrides
• No merge ambiguity

---

User Override Model

To override Azure defaults, users must either:

1. Disable the corresponding *Enabled knob, or
2. Provide a higher‑priority drop‑in at a different path (e.g., 90-user.conf)

---

Rationale

• Clear ownership of cloud-managed paths
• Deterministic behavior
• Avoid reliance on Ignition merge quirks or more complex merge/conflict strategies
• Fail‑fast detection of misconfiguration
• Easy to understand and audit
• Maintains configuration comparable to that supported by WALinuxAgent today, while also providing knobs to control behaviors
• Extensions. would enable extensibility for knobs desired in the future (say to manage behaviors required for MSP, CVM encryption, IMDS retries)

[travier]

This issue was discussed in the community meeting on 2026-01-14.

[travier]

If we start adding cloud specific extensions then that kind of defeats the purpose of Ignition being a platform agnostic way to configure systems.

Part of the rationale for Ignition is to either produce a configured system as request or no machine at all so we can not fail and continue.

From https://coreos.github.io/afterburn/platforms/, it looks like Afterburn already supports setting up SSH keys for users on Azure. Why should we add a new way to do that?

This is also trying to do a lot of things at the same time. For example, from coreos/afterburn#1242 (comment), I think you should focus on the "reporting status" part as a first self-contained feature.

[cjp256]

If we start adding cloud specific extensions then that kind of defeats the purpose of Ignition being a platform agnostic way to configure systems.

Part of the rationale for Ignition is to either produce a configured system as request or no machine at all so we can not fail and continue.

Love that statement.

I think it's all perspective.  My perspective is that the "machine" comprises both Ignition config provided by the user via user/custom-data and the config provided by the user to Azure for provisioning. This is a proposal to meld the two in a way that is extensible and comprehensive.

From https://coreos.github.io/afterburn/platforms/, it looks like Afterburn already supports setting up SSH keys for users on Azure. Why should we add a new way to do that?

Afterburn's ssh-keys feature appears largely oriented around 'core' user, or at least requires some other orchestrator to use it for additional accounts.  Flatcar doesn't use it, at least on Azure.  I'm not sure how it is used on other images if not strictly to finish 'core' configuration, which I want to emphasize is not the same as the Azure user created by WALinuxAgent (and being proposed here).

This is also trying to do a lot of things at the same time. For example, from coreos/afterburn#1242 (comment), I think you should focus on the "reporting status" part as a first self-contained feature.

Perhaps, but I think it's important that we sketch out a design from the top that works for everyone. I know there's been some churn on this, but that's largely because the work was exploratory and trying to understand ignition, afterburn, flatcar, etc. We're learning still! :D

Ignition:

• is early in boot, great time to lay down configuration
• supports user creation. Afterburn would need logic to create users and configure them. Note that WALinuxAgent is creating the user and configuring the ssh keys today based on VM's osProfile configuration.
• supports disk setup. Afterburn would need logic to configure disks and mount them (or systemd units to do so).
• mounts the provisioning media (so we have access to necessary configuration).  Afterburn would need to assume a dependency that Ignition mounted it, or would have to remount it, or Ignition would have to copy it somewhere for Afterburn to consume.
• fetches user-data, it would be slightly more optimal to just make one request for all the instance metadata than retrieve it in multiple steps.

fwiw we do have azure-init, which essentially does all this already (creates user, configures ssh keys, configures drop-ins, reports ready/failure).  Trying to do this all is in Afterburn would essentially be a copy/paste and refactor of 90% of it.  I'm not sure it makes practical sense to maintain both.  Regardless, neither azure-init or Afterburn solve the fundamental problems of changes to fetch/decrypt custom data and validation of user-data for Confidential VM scenarios, etc.  So even if we just suggested including azure-init in Azure images, we will need to make more extensive changes to Ignition to support these scenarios at some point (unless they're not desired).

[travier]

I'm not asking you to put everything back in Afterburn. I'm suggesting you start with integrating progress reporting in Ignition as this would be a feature specific to a cloud provider but it would not be exposed to users in the config so the bar for getting this accepted would be much lower.

[travier]

Why are you mentioning disk setup? I don't see anything disk related in #2185

[travier]

If I understand #2185 correctly, this also adds a new source to read to get the config (the OVF Provisioning Data from a disk). The current implementation tries the IMDS first then falls back to the disk to get the config: https://github.com/coreos/ignition/blob/main/internal/providers/azure/azure.go.

So if we want to generate configs out of the one on the disk then it means that we must wait for it to come up (is it guaranteed to always be there?).

[travier]

Is there documentation somewhere about what each of those files in the attach disk drive includes and the expected format?

[cjp256]

Sorry I realized last night I did neglect these pertinent details and you beat me back here :D

The provisioning media for Linux VMs contains two files: ovf-env.xml and CustomData.bin.

On Azure, user-data and custom-data are often used interchangeably, but they differ. Custom data is only available during provisioning via the attached UDF media that Ignition mounts, and the media is ejected once the VM reports ready or failure. User-data, by contrast, is always accessible through IMDS. Because IMDS is easy to access, Azure discourages passing sensitive data via user-data; custom-data has additional platform protections and is only available during provisioning. This is why ignition checks for custom data if no user data is found.

ovf-env.xml is the primary source of VM configuration. It contains nearly all parameters provided at VM creation, including custom data and admin password, but it does not have user-data. Cloud-init parses ovf-env.xml here. Likewise IMDS offers most everything except admin password and custom data.

Ignition is one of the few distros that rely on CustomData.bin. The proposed PR change migrates to using ovf-env.xml instead, allowing access to the full remaining configuration in a single place.

We appreciate your time and effort thinking about this @travier !

*edited for clarity

[dustymabe]

We had a video community meeting on this topic a few weeks back:

• recording
• notes