Continuous reconcile loop (~1/sec) on Console and User controllers after deploying standalone Console CR

[dvdlevanon]

Environment

redpanda-operator: v25.3.1
redpanda: v25.3.10
console (standalone CRD): v3.3.2
Kubernetes: EKS

Describe the bug

After deploying a standalone Console CR against a healthy Redpanda cluster, both the console and user controllers enter a continuous reconcile loop that never settles — firing every ~1 second indefinitely.
Before deploying the Console CR the cluster was fully stable with no reconcile activity.

{"level":"debug","ts":"2026-03-10T13:11:39.905Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"0263b5ef-f947-465c-9989-9fdff004dc77"}
{"level":"info","ts":"2026-03-10T13:11:39.905Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"1f58b82c-4394-47ab-9bd4-67c01ae3739d","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:40.199Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"0263b5ef-f947-465c-9989-9fdff004dc77","elapsed":0.294841631}
{"level":"debug","ts":"2026-03-10T13:11:41.101Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"70e9b541-f7fc-447d-9007-da9675159cc3"}
{"level":"info","ts":"2026-03-10T13:11:41.101Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"57854ad8-66d4-4c99-a130-844fe6eb9655","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:41.389Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"70e9b541-f7fc-447d-9007-da9675159cc3","elapsed":0.287974374}
{"level":"info","ts":"2026-03-10T13:11:43.376Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"7951309a-9e77-476f-9654-186942d3f811","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:43.376Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"a86ef417-a7c2-4ad9-b715-14d191125099"}
{"level":"debug","ts":"2026-03-10T13:11:43.691Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"a86ef417-a7c2-4ad9-b715-14d191125099","elapsed":0.314965255}
{"level":"info","ts":"2026-03-10T13:11:44.891Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"ece2900d-cc64-4378-8424-ea51dbe4ef9d","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:44.892Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"0a4927e2-0c0d-4194-ac02-c6dcf9dac7f4"}
{"level":"debug","ts":"2026-03-10T13:11:45.273Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"0a4927e2-0c0d-4194-ac02-c6dcf9dac7f4","elapsed":0.380318821}
{"level":"debug","ts":"2026-03-10T13:11:46.415Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"28ecdcfe-e459-4a37-80a6-ee07ae77426a"}
{"level":"info","ts":"2026-03-10T13:11:46.415Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"c80bc5af-3e35-4c55-86d6-9de66affd228","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:46.765Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"28ecdcfe-e459-4a37-80a6-ee07ae77426a","elapsed":0.349423179}
{"level":"debug","ts":"2026-03-10T13:11:47.874Z","logger":"UserReconciler.Reconcile","msg":"Starting reconcile loop","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"1a7d6cb4-b5a5-4d83-a2e4-57b08f271d4a"}
{"level":"info","ts":"2026-03-10T13:11:47.874Z","msg":"reconciling Console","controller":"console","controllerGroup":"cluster.redpanda.com","controllerKind":"Console","Console":{"name":"redpanda","namespace":"redpanda"},"namespace":"redpanda","name":"redpanda","reconcileID":"f6af59cf-5df2-43b2-9caa-2cd222ba200c","key":{"name":"redpanda","namespace":"redpanda"}}
{"level":"debug","ts":"2026-03-10T13:11:48.186Z","logger":"UserReconciler.Reconcile","msg":"Finished reconciling","controller":"user","controllerGroup":"cluster.redpanda.com","controllerKind":"User","User":{"name":"team-admin","namespace":"redpanda"},"namespace":"redpanda","name":"team-admin","reconcileID":"1a7d6cb4-b5a5-4d83-a2e4-57b08f271d4a","elapsed":0.312383105}

Symptoms

• "reconciling Console" fires every ~1 second non-stop
• User controller loops continuously as a side effect
• Redpanda CRD resourceVersion increments every ~1 seconds despite no spec changes
• Console CRD resourceVersion is completely stable
• Cluster is fully healthy — all conditions Ready, Healthy, Stable
• Deleting the Console CR stops the loop

Expected behavior
Once the cluster reaches a stable state, reconcile activity should settle. Controllers should not loop continuously when nothing has changed.