[FR] multicluster operator: wire the decommissioning controller (auto-PVC cleanup after broker decom)

[david-yu]

Summary

The single-cluster operator binary (operator/cmd/run/run.go) ships two storage-cleanup controllers:

• nodewatcher → pvcunbinder.Controller (deletes PVCs of pods stuck Pending due to Node deletion)
• decommissioning → decommissioning.Controller (handles broker decom + storage cleanup)

The multicluster operator binary (operator/cmd/multicluster/multicluster.go) only wires pvcunbinder.MulticlusterController. The decommissioning controller is not registered, so auto-decommission events leave behind orphaned PVCs (and, with reclaimPolicy: Delete storage classes, mostly-orphaned managed disks once the StatefulSet eventually shrinks too).

This feature request asks for the decommissioning controller to be wired into the multicluster operator the same way it is in the single-cluster operator, so that StretchCluster + NodePool deployments can rely on the operator for the same broker-lifecycle storage cleanup that single-cluster Redpanda chart users get out of the box.

Why it matters

Concrete failure mode we hit running https://github.com/david-yu/redpanda-operator-stretch-beta's Demo B (regional failure + failover-region capacity injection):

1. Region down → cluster auto-decommissions broker IDs 0, 1 (their replicas drained onto the failover region's brokers).
2. Region restored (via az aks stop / start, EC2 instance recovery, etc.). Node objects come back, the StatefulSet schedules pods on the new nodes, and the pods bind to the same datadir-redpanda-rp-east-{0,1} PVCs because Azure managed disks survive aks stop intact.
3. The redpanda containers start, find their old node_uuid in /var/lib/redpanda/data, and try to rejoin as the previously-decommissioned IDs. The cluster rejects them:

   bad_rejoin: trying to rejoin with same ID and UUID as a decommissioned node
   

   Pods then loop indefinitely at 1/2 Running.

PVCUnbinder doesn't help here — it's keyed on pods stuck Pending (Node deletion + nodeAffinity break), but in aks stop/start the disks survive and the new pods immediately bind, so they never sit Pending. They just restart-loop on bad_rejoin.

The single-cluster decommissioning controller is the right component to handle this: when a broker is decommissioned (either by the cluster's own partition autobalancer, or by rpk redpanda admin brokers decommission), it should delete the broker's PVC. With reclaimPolicy: Delete on the StorageClass, the disk is reaped, and the next time the StatefulSet creates a pod with that ordinal, it gets a fresh PVC + disk + node UUID and joins as a new broker ID.

Workaround today

Manual two-step recovery whenever a region restores after auto-decom:

kubectl --context rp-<lost-region> -n redpanda delete pvc datadir-redpanda-<pool>-<id>
kubectl --context rp-<lost-region> -n redpanda delete pod redpanda-<pool>-<id> --grace-period=0 --force

Operationally fragile and not how the single-cluster path documents it.

Proposed change

Wire the existing operator/internal/controller/decommissioning/ controller into operator/cmd/multicluster/multicluster.go alongside the PVCUnbinder, with the same flag surface used by the single-cluster path (--additional-controllers=decommission style, or unconditionally on, matching whatever the rest of the multicluster controllers do). The chart's additionalCmdFlags already passes through, so users can opt in via helm values.

Specifically:

• cmd/multicluster/multicluster.go: add an import of operator/internal/controller/decommissioning and a SetupWithMultiClusterManager (or equivalent) call alongside the existing PVCUnbinder block.
• Multicluster RBAC: extend the operator's ClusterRole/Role to include the same verbs the single-cluster decommissioning controller needs on PVCs and PVs (the chart already has pvcunbinder.ClusterRole.yaml; sibling decommission.ClusterRole.yaml exists for the single-cluster path and should mirror).

If there's a reason the decommissioning controller isn't multicluster-safe today (e.g. it assumes a single-cluster Redpanda CR, not a StretchCluster + NodePool pair), happy to scope the work — would be useful to know what blockers exist.

Environment

• Operator chart v26.2.1-beta.1 (multicluster build)
• Redpanda v26.1.6
• Validated end-to-end on AKS (eastus / westus2 / centralus + eastus2 failover) and discussed in multicluster operator: NodePool / StretchCluster controllers don't reconcile on a newly-added peer until the operator Deployment is restarted #1493 (which is a separate reconciler-bootstrap bug, not the same root cause)

Repo with full repro: https://github.com/david-yu/redpanda-operator-stretch-beta — Demo B exercises the auto-decommission + region restore flow.

[david-yu]

Reproduced on operator v26.2.1-beta.1 (multicluster build) — 2026-05-02

Confirming this is still open. Full validation just hit it on the same scaffold the issue references (david-yu/redpanda-operator-stretch-beta) running Demo B end-to-end on AWS.

Setup

• 3-region StretchCluster brought up clean (rp-east in us-east-1, rp-west in us-east-2 wait — actually us-west-2, rp-eu in eu-west-1) — 5 brokers, RF=5 topics, Healthy: true.
• Failover region (rp-failover in us-east-2) added per Demo B Step 3-4: 7 brokers total, PEERS=4 / UNHEALTHY=0 from rpk k8s multicluster status.

Trigger

• Cordon all rp-east nodes + delete brokers 0+1 (Demo B Step 1).
• After capacity-injection + controller-leadership transfer to a us-east-2 broker (Demo B AWS-only step in Step 4 — required because the cross-Atlantic heartbeat hits the hardcoded 100ms node_status_rpc timeout), manually decommission brokers 0 and 1:

  rpk redpanda admin brokers decommission 0 --skip-liveness-check
  rpk redpanda admin brokers decommission 1 --skip-liveness-check

• Decom completes — rpk cluster health reports All nodes: [2 3 4 5 6], Nodes down: [], Healthy: true. Brokers 0 and 1 are out of the cluster.

Observation 1 — orphaned PVCs (the issue body's primary claim)

$ kubectl --context rp-east -n redpanda get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-redpanda-rp-east-0   Bound    pvc-28600af7-997c-48d9-afff-6e1269c25cb0   20Gi       RWO            gp2            15h
datadir-redpanda-rp-east-1   Bound    pvc-2f439d12-4885-497d-a526-550953e7a87f   20Gi       RWO            gp2            15h

Both PVCs still Bound 15h after the brokers were decommissioned out of the cluster. The multicluster operator did nothing — exactly what the issue says.

Observation 2 — multiple leaked EBS volumes

$ aws ec2 describe-volumes --region us-east-1 \
  --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=datadir-redpanda-rp-east-0,datadir-redpanda-rp-east-1" \
  --query 'Volumes[].[VolumeId,State,Tags[?Key==`kubernetes.io/created-for/pvc/name`]|[0].Value]' --output text
vol-098aeb31b04c2bcb2  available  datadir-redpanda-rp-east-0
vol-01f15ea7799d2d3de  available  datadir-redpanda-rp-east-1
vol-0556a885041d7bda0  available  datadir-redpanda-rp-east-1
vol-01fe14b52599b7071  available  datadir-redpanda-rp-east-0
vol-09239458c145a16ee  available  datadir-redpanda-rp-east-0

5 available (= unattached) volumes for two PVC names — all from earlier broker recreations during the validation run that themselves stayed orphaned. With reclaimPolicy: Delete set on the StorageClass these would normally be reaped on PVC delete, but since no PVC delete ever happens, they all sit and accrue cost.

Observation 3 — variant on the post-restore behavior

The issue body describes the brokers crash-looping on bad_rejoin: trying to rejoin with same ID and UUID as a decommissioned node once the region is restored. In our run the broker container itself stays Ready=true after uncordon (Redpanda starts up fine reading the persisted state), but:

• The sidecar container is Ready=false indefinitely. Its post-start lifecycle hook is calling DELETE /v1/brokers/0/maintenance which 404s with broker with id 0 not found — because from the controller's perspective broker 0 is gone, even though the local pod thinks it's still id 0.
• Cluster state diverges depending on which view you query:

  $ rpk redpanda admin brokers list   # local broker's view
  ID    HOST                          MEMBERSHIP   IS-ALIVE  UUID
  0     redpanda-rp-east-0.redpanda   active       true      56d622b9-…
  1     redpanda-rp-east-1.redpanda   active       true      fafa6bec-…
  2     redpanda-rp-eu-0.redpanda     active       true      e9966939-…
  ...
  
  $ rpk cluster health                 # controller's authoritative view
  Healthy:    true
  All nodes:  [2 3 4 5 6]              # 0 and 1 are gone

So instead of the bad-rejoin crash loop, we get a quiet split-brain: pods are 1/2 Ready (looking healthy at first glance) but the redpanda broker process is running with a node_uuid the controller has rejected. The same workaround the issue documents (manually kubectl delete pvc + pod --force so the StatefulSet rebuilds with fresh storage) recovers it.

Workaround applied (manual, what the decommissioning controller would automate):

kubectl --context rp-east -n redpanda delete pvc datadir-redpanda-rp-east-0 datadir-redpanda-rp-east-1
kubectl --context rp-east -n redpanda delete pod redpanda-rp-east-0 redpanda-rp-east-1 --grace-period=0 --force

After this, the StatefulSet recreates with fresh PVCs/disks, brokers boot cold and join as new IDs (7, 8) cleanly.

Environment

• Operator: redpanda-data/operator @ 26.2.1-beta.1 (multicluster build), helm chart 26.2.1-beta.1.
• Broker: redpandadata/redpanda:v26.1.6.
• Kubernetes: EKS 1.34 (us-east-1, us-west-2, eu-west-1, us-east-2).
• StorageClass: gp2 (reclaimPolicy: Delete, default per the same-cloud beta's terraform).
• Cluster: 4-region StretchCluster + NodePool, RF=5, flat cross-cluster networking.

The proposed fix (wiring decommissioning.Controller into cmd/multicluster/multicluster.go alongside pvcunbinder.MulticlusterController) would make this self-recovering — happy to test a build that has it.