CNV-67916: hypershift, kubevirt, add enhancement for IP management #1855

qinqon · 2025-10-07T10:19:52Z

This enhancement proposes adding static IP address management (IPAM) capabilities to HyperShift when using the KubeVirt provider and the KubeVirt nodepool is not attached to default pod network using a multus layer2 network to replace it.

The implementation enables operators to define IP pools and network configurations that are automatically allocated to virtual machines during cluster provisioning. This functionality addresses the need for predictable, static network configuration in environments where DHCP is not available or desirable, particularly in on-premises and edge deployments.

https://issues.redhat.com/browse/CNV-67916

openshift-ci · 2025-10-07T10:20:06Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign derekwaynecarr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

enhancements/hypershift/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-10-07T10:23:24Z

@qinqon: This pull request references CNV-67916 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.21.0" version, but no target version was set.

In response to this:

This enhancement proposes adding static IP address management (IPAM) capabilities to HyperShift when using the KubeVirt provider and the KubeVirt nodepool is not attached to default pod network using a multus layer2 network to replace it.

The implementation enables operators to define IP pools and network configurations that are automatically allocated to virtual machines during cluster provisioning. This functionality addresses the need for predictable, static network configuration in environments where DHCP is not available or desirable, particularly in on-premises and edge deployments.

https://issues.redhat.com/browse/CNV-67916

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

enhancements/hypershift/kubevirt-static-ip-address-management.md

qinqon · 2025-10-14T15:26:46Z

/cc @maiqueb

qinqon · 2025-10-15T07:15:12Z

/cc @orenc1

celebdor · 2025-10-27T09:25:24Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+    // Addresses specify IP ranges from which addresses will be allocated for this interface.
+    // Supports CIDR notation, hyphenated ranges, and single IPs.
+    // +required
+	Addresses []string `json:"addresses"`


This should probably use hostedcluster_types things like CIDRBlock or ipNet, or maybe define a new one that can be CEL validated if it is really necessary to have ranges and single IPs

This should probably use hostedcluster_types things like CIDRBlock or ipNet, or maybe define a new one that can be CEL validated if it is really necessary to have ranges and single IPs

Since this can be a cidr, ip range or single IP I cannot use those types, so this has to be a string slice, I will add a CEL validation.

celebdor · 2025-10-27T09:30:27Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+
+**Management Cluster Components**:
+- Cluster API Provider KubeVirt controller (KubevirtMachineTemplate controller): Enhanced with IP allocation logic
+- HyperShift operator (NodePool controller): Updated to handle network configuration in NodePool spec


Won't it also be necessary to update the ignition generation for the the netconfig?

Won't it also be necessary to update the ignition generation for the the netconfig?

The fact that we implmenet "IP allocation logic" using virtual machine cloud init network data it's implementation details, so it's part of the Cluster API Provider Kubevirt controller changes

celebdor · 2025-10-27T09:32:42Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+
+- Adds complexity to the hypershift kubevirt controllers:
+  - implementing and ip pool mechanism
+  - Needs to parse and understand the openshift network config format


s/openshift/openstack/ ?

s/openshift/openstack/ ?

Right, I will also add the it should unerstand either openstack or netplan depending of the kind of networkData we want to use either configdrive or nocloud

celebdor · 2025-10-27T09:34:01Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+- Symptom: New KubevirtMachine resources remain in Pending state
+- Detection: Check KubevirtMachine events and status conditions
+- Log output: "No available IPs in pool for interface X"
+- Metric: `kubevirt_ippool_exhausted{cluster="X"}`


Is this going to be a status condition on the NodePool?

Is this going to be a status condition on the NodePool?

It should, I will add that

maiqueb

Thank you.

maiqueb · 2025-10-27T08:31:33Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+
+- Enable static IP address assignment for HyperShift guest cluster nodes running on KubeVirt
+- Provide a flexible API for defining IP address pools with support for various IP range formats (CIDR, hyphenated ranges, single IPs)
+- Support both IPv4 and IPv6 addressing


so the kubevirt VMs which are actually the hosted cluster nodes can be ipv6 nodes ?

do we support single stack ipv6 ? or only dual-stack ?

Just trying to figure out the test matrix we need to have later on.

so the kubevirt VMs which are actually the hosted cluster nodes can be ipv6 nodes ?

do we support single stack ipv6 ? or only dual-stack ?

Just trying to figure out the test matrix we need to have later on.

For hyperhsift kubevirt we don't use KubeVirt DHCP mechanism we use OVN DHCPOptions instead this allow us to support single stack ipv4/ipv6 and dual stack.

but isn't this about localnet secondary interfaces ?

AFAICT we do not have OVN DHCPOpts for localnet secondaries - only on the pod network (masquerade or primary UDN attachments).

but isn't this about localnet secondary interfaces ?

AFAICT we do not have OVN DHCPOpts for localnet secondaries - only on the pod network (masquerade or primary UDN attachments).

Right, I mixed things up, so since this is static IP assignment and we don't depend on KubeVirt DHCP capabilities we support sintle stack (both ipv4 and ipv6) and dual stack.

OK, that's aligned with my expectations.

OK, so the test matrix will be complex. Do we need to have anything in origin ?...

OK, that's aligned with my expectations.

OK, so the test matrix will be complex. Do we need to have anything in origin ?...

At origin we test dual stack

enhancements/hypershift/kubevirt-static-ip-address-management.md

maiqueb · 2025-10-27T10:09:58Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+Migrating nodepools nodes from dynamic to static IPs is not expected, those nodes
+will be reconstructed and new ip assigned.


I'm sorry, but I don't follow this.

I'm sorry, but I don't follow this.

I will reword this

maiqueb · 2025-10-27T10:13:12Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+- BootstrapNetworkConfig in KubevirtMachineSpec (cluster-api-provider-kubevirt)
+- BootstrapNetworkConfig in nodepool's KubevirtPlatformSpec (hypershift)


I'm forced to say one of the most confusing things to me (I'm not HCP savvy) was to differentiate between these two CRDs.

When I would - as a hosted cluster "owner" - use one of these versus the other ?

Which of these should I use and why ? Or do I need to use both ?

I'm forced to say one of the most confusing things to me (I'm not HCP savvy) was to differentiate between these two CRDs.

When I would - as a hosted cluster "owner" - use one of these versus the other ?

Which of these should I use and why ? Or do I need to use both ?

At hypershift you use the NodePool CRD, KubevirtMachine is internal to hypershift.

maiqueb · 2025-10-27T10:14:46Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+1. **IP Pool Exhaustion**: Machine creation fails with clear error. Operator must expand pool or delete unused machines.
+2. **Invalid Network Configuration**: HostedCluster creation fails validation. Operator must correct configuration.
+3. **Network Config Application Failure**: Node fails to boot or is unreachable. Visible in VM console logs. Operator must verify network configuration correctness.
+4. **IP Allocation Conflict**: Different nodepools using same network but with overlapping IPs should fail with clear error. Operator should fix the subnets.


why should this matter ? The way I see it, this would become an issue if the node pools are being used at the same time in the same node

why should this matter ? The way I see it, this would become an issue if the node pools are being used at the same time in the same node

Nodepools cannot be used for the same node, since they are definining those nodes, issue is that one nodepool can contribute to the ip exhaustion of the other if they have overlapping subnets.

what I don't follow is how can one network have multiple node pools.

Can you point me towards documentation about the node pool ? What does it reflect, and how it relates to the network ?

So the nodepool kubevirt specific api hass the additionalNetworks field, to specify the NAD name, this enhancement is adding the IPAM configuration at same API level, and a hosted cluster can have multiple nodepools, this means multiple kubevirt nodepools can have conflicting IPAM configuration, and that's what we want to detect.

maiqueb · 2025-10-27T10:18:06Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+- Resolution: Verify OpenStack network config format correctness, check gateway/DNS reachability
+
+**IP Allocation Errors**:
+- Symptom: Machine creation fails or succeeds with incorrect IP


is it realistic to include incorrect IP allocation in this metric ?

Asking because (in my own limited mental model) we could prevent this, instead of capturing it in a metric.

Couldn't we ?

is it realistic to include incorrect IP allocation in this metric ?

Asking because (in my own limited mental model) we could prevent this, instead of capturing it in a metric.

Couldn't we ?

Agree since we are already using CEL to ensure propre IPs are configured at the addresses

is it realistic to include incorrect IP allocation in this metric ?

Asking because (in my own limited mental model) we could prevent this, instead of capturing it in a metric.

Couldn't we ?

Agree since we are already using CEL to ensure propre IPs are configured at the addresses, I will remove it

maiqueb · 2025-10-27T10:21:17Z

enhancements/hypershift/kubevirt-static-ip-address-management.md

+- New nodes will use DHCP
+- No impact on running workloads


not so fast - I'm not so sure this is this easy / there aren't impacts.

Let's say you're using static IPs, then you stop using it. You remove the NetworkConfig from the template spec.

Your existing nodes with static IPs will continue to function.

a new node will gets its IP via DHCP

what will prevent this new node from getting an IP which is already in use by the static IPs counter part ?

AFAIU, you don't have anything to map the static IPs from the node pool to the DHCP pool availability.

not so fast - I'm not so sure this is this easy / there aren't impacts.

Let's say you're using static IPs, then you stop using it. You remove the NetworkConfig from the template spec.

Your existing nodes with static IPs will continue to function.

a new node will gets its IP via DHCP

what will prevent this new node from getting an IP which is already in use by the static IPs counter part ?

AFAIU, you don't have anything to map the static IPs from the node pool to the DHCP pool availability.

Even though I think we should not support this, it may work if NodePool is changed so bootstrap network config is removed and virtual machines are restarted.

Signed-off-by: Enrique Llorente <ellorent@redhat.com>

openshift-ci · 2025-11-03T14:28:03Z

@qinqon: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/markdownlint	`8287292`	link	true	`/test markdownlint`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from derekwaynecarr and sjenning October 7, 2025 10:20

qinqon changed the title ~~hypershift, kubevirt: Add enhancement for IP management~~ CNV-67916: hypershift, kubevirt, add enhancement for IP management Oct 7, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 7, 2025

phoracek suggested changes Oct 7, 2025

View reviewed changes

qinqon force-pushed the hypershift-kubevirt-ipam branch 4 times, most recently from b8d3daf to b6cb5a1 Compare October 9, 2025 09:08

qinqon requested a review from phoracek October 9, 2025 09:16

openshift-ci bot requested a review from maiqueb October 14, 2025 15:26

openshift-ci bot requested a review from orenc1 October 15, 2025 07:15

celebdor suggested changes Oct 27, 2025

View reviewed changes

maiqueb reviewed Oct 27, 2025

View reviewed changes

qinqon force-pushed the hypershift-kubevirt-ipam branch 2 times, most recently from 908013f to 5db1dd5 Compare October 28, 2025 13:59

qinqon requested review from celebdor and maiqueb October 28, 2025 14:01

qinqon force-pushed the hypershift-kubevirt-ipam branch 4 times, most recently from a60fca8 to c7dcbdb Compare November 3, 2025 14:03

hypershift, kubevirt: Add enhancement for IP management

8287292

Signed-off-by: Enrique Llorente <ellorent@redhat.com>

qinqon force-pushed the hypershift-kubevirt-ipam branch from c7dcbdb to 8287292 Compare November 3, 2025 14:04

		Migrating nodepools nodes from dynamic to static IPs is not expected, those nodes
		will be reconstructed and new ip assigned.

		- BootstrapNetworkConfig in KubevirtMachineSpec (cluster-api-provider-kubevirt)
		- BootstrapNetworkConfig in nodepool's KubevirtPlatformSpec (hypershift)

CNV-67916: hypershift, kubevirt, add enhancement for IP management #1855

Are you sure you want to change the base?

CNV-67916: hypershift, kubevirt, add enhancement for IP management #1855

Uh oh!

Conversation

qinqon commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 7, 2025

Uh oh!

openshift-ci-robot commented Oct 7, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qinqon commented Oct 14, 2025

Uh oh!

qinqon commented Oct 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maiqueb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qinqon commented Oct 7, 2025 •

edited

Loading

openshift-ci-robot commented Oct 7, 2025 •

edited by openshift-ci bot

Loading