StretchCluster: add ghost node ejection acceptance test

[hidalgopl]

Summary

Adds a multicluster acceptance test that verifies Redpanda's ghost node ejection — the automatic decommission of a broker whose region becomes permanently unreachable. Deletes the k3d agent hosting one region and asserts that the surviving regions' rpk cluster health eventually reports 2 of 3 brokers.

What's in the change

• New feature: stretch-cluster-ghost-node-ejection.feature — 3-region stretch cluster with aggressive autobalancing timeouts (avail=30s, auto-decom=60s) on Redpanda v26.1.5. Tagged @serial since it destroys infrastructure.
• New steps (stretch_ghost_node_ejection.go): discover the controller region via rpk, pick a non-controller region to take offline via k3d node delete, and poll the remaining regions for the reduced node count. Cleanup deletes the stale NotReady Node object.
• Framework changes (stretch.go): pin each vcluster's workloads to a single k3d agent via sync.fromHost.nodes.selector.labels: kubernetes.io/hostname + virtual scheduler. Needed so k3d node delete atomically removes the broker — a partial outage doesn't trigger Redpanda's auto-decommission path in our vcluster setup.
• Supporting: pre-pull v26.1.5, loft-sh/kubernetes:v1.33.4, and cert-manager v1.17.1 images in the Taskfile.

🤖 Generated with Claude Code

[andrewstucki]

So the Github UI is having a hard time loading the new steps file for me, but I took a look directly in your branch. A few things:

1. Would it be possible to just simulate a regional outage via disconnecting the vcluster instance and killing the broker pods like I did in the region-killing test? That would make it so that you could run this test in parallel with the other tests rather than having to physically tear down a worker node.
2. If you still want to do the worker node teardown, I believe there are a bunch of helpers in the harpoon framework for doing so without issuing k3d commands directly in case we ever get around to implementing the providers for running the tests against real cloud infrastructure. Take a look at some of the pvc unbinder tests that leverage the same sort of behavior and re-use what you can from there if need be.

[andrewstucki]

Github is having issues with me trying to edit my comments as well, but we could also potentially just merge this test with the regional outage test if we want -- just add the configuration for ejection and the verification step before bringing the region back online.

[joe-redpanda]

Be careful scaling these parameters. partition balancer is an old component that makes a lot of implicit assumptions regarding the relative scale of these parameters.

I crystallized the guidance in validators.cc which is as follows:

Rules:

node_status_interval must be less than
partition_autobalancing_tick_interval_ms

node_status_interval * 7 must be less than
partition_autobalancing_node_availability_timeout_sec

health_monitor_tick_interval must be less than
partition_autobalancing_tick_interval_ms

partition_autobalancing_node_availability_timeout_sec must be less than
partition_autobalancing_node_autodecommission_timeout_sec

my recommendation is that if you're going to scale one of the parameters, try to scale all of them accoringly

for your situation:
have node status be 5 seconds or so
have partition balancer tick interval be 10s or so
have node availability be 45 seconds or so
have node autodecommission be 90s or so

Total ejection time is still 90s here, because the underlying logic doesn't check node unavailable and then start the timer on auto ejection, its just 90s after a quorum agrees that the node has been missing too long.

[hidalgopl]

thanks! adjusted parameters as you suggested

[hidalgopl]

So the Github UI is having a hard time loading the new steps file for me, but I took a look directly in your branch. A few things:

1. Would it be possible to just simulate a regional outage via disconnecting the vcluster instance and killing the broker pods like I did in the region-killing test? That would make it so that you could run this test in parallel with the other tests rather than having to physically tear down a worker node.
2. If you still want to do the worker node teardown, I believe there are a bunch of helpers in the harpoon framework for doing so without issuing k3d commands directly in case we ever get around to implementing the providers for running the tests against real cloud infrastructure. Take a look at some of the pvc unbinder tests that leverage the same sort of behavior and re-use what you can from there if need be.

1. Yes, I tried that initially, but couldn't make the decommission work this way. I suspect it is due to k8s service for broker being still there and having publishNotReadyAddresses: true, but didn't dig deeper into it. Tried various ways to make it work without removing k3s node, but none of them worked for me.

2. Good point, I made changes to use harpoon's helpers. Also, decided not to re-add the node back for now. I marked the test as @serial with an assumption that it will run as the last one in the suite. Initially I thought about re-adding it, but this would also require loading all docker images on the node again + potentially re-deploying the operator / vCluster there. I believe the most pragmatic solution for now would be just not re-adding it and using @serial.

3. I also tried to merge it with regional outage test - however:
   a) The autodecommission doesn't work with the current takeRegionDown implementation.
   b) If we would change the takeRegionDown to remove k3s node there, it means that later on we'd need to re-add and re-deploy everything, which adds up a lot of time to the suite.

cc @andrewstucki

[andrewstucki]

LGTM for now -- eventually we may want to remove the "k3dNode" field naming, etc. from this rather than just "nodeName" since we're technically not guaranteed to be running this against k3d, but I'm fine if we change that later, if you want to, feel free to merge or just swap the field name and then merge.