Skip to content

fix(controller): prevent NodeAgent restarts from ESO metadata updates #1999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Nov 14, 2025
+540 −0
Merged

fix(controller): prevent NodeAgent restarts from ESO metadata updates #1999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions internal/controller/dataprotectionapplication_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package controller
import (
"context"
"os"
"reflect"

"github.com/go-logr/logr"
routev1 "github.com/openshift/api/route/v1"
Expand Down Expand Up @@ -193,6 +194,21 @@ func (l *labelHandler) Update(ctx context.Context, evt event.TypedUpdateEvent[cl
if evt.ObjectNew.GetLabels()[oadpv1alpha1.OadpOperatorLabel] == "" || dpaname == "" {
return
}

// For Secrets, check if only metadata changed (e.g., ResourceVersion updates from ESO)
// This prevents unnecessary reconciliations when external-secrets-operator updates metadata
if oldSecret, ok := evt.ObjectOld.(*corev1.Secret); ok {
if newSecret, ok := evt.ObjectNew.(*corev1.Secret); ok {
// Skip reconciliation if data, stringData, and labels haven't changed
// This filters out metadata-only updates (ResourceVersion, ManagedFields, etc.)
if reflect.DeepEqual(oldSecret.Data, newSecret.Data) &&
reflect.DeepEqual(oldSecret.StringData, newSecret.StringData) &&
reflect.DeepEqual(oldSecret.Labels, newSecret.Labels) {
Copy link
Member Author

@kaovilai kaovilai Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weshayutin although.. do we know if they are changing label every 30 seconds? are we supposed to ignore label changes?

Copy link
Contributor

@weshayutin weshayutin Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple notes that I have would be.

  1. I could be wrong but it's my understanding the external secrets operator is tech-preview.
  2. I'm not the smartest guy in the world but it seems to me that ESO should check the secret but if no change is required, I think ANY metadata on the secret should remain UNCHANGED, hence why I moved the bug to ESO.
  3. I DO like being defensive in this case, and we either need to set it up and try/test or enquire w/ the ESO team re: labels.
  4. I don't think this is a priority for our attention atm based on my above understanding. I could be wrong though.

Copy link
Member

@shubham-pampattiwar shubham-pampattiwar Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also check for annotations ?

Copy link
Member Author

@kaovilai kaovilai Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are explicitly avoiding reconcile on annotation changes.

Copy link
Member Author

@kaovilai kaovilai Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eg. avoiding reconcile on https://github.com/external-secrets/external-secrets/blob/579135b5bb558b8b79f71bfc57f1a852968abac4/docs/provider/ibm-secrets-manager.md?plain=1#L283-L284

return
}
}
}
Comment on lines +197 to +210
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add Secret.Type to the comparison; StringData check is ineffective in practice.

The metadata-only filter correctly prevents ESO-triggered reconciliations, but:

  1. Secret.Type is not checked: If the Secret type changes (e.g., from Opaque to kubernetes.io/tls), this should trigger reconciliation, but currently won't.

  2. StringData is always empty when reading Secrets: Kubernetes converts StringData to Data on write and returns empty StringData on read. Both oldSecret.StringData and newSecret.StringData will typically be empty in real scenarios, making this check always pass. While harmless, it doesn't provide the intended protection and could be misleading.

Apply this diff to add the Type field check:

 	// For Secrets, check if only metadata changed (e.g., ResourceVersion updates from ESO)
 	// This prevents unnecessary reconciliations when external-secrets-operator updates metadata
 	if oldSecret, ok := evt.ObjectOld.(*corev1.Secret); ok {
 		if newSecret, ok := evt.ObjectNew.(*corev1.Secret); ok {
 			// Skip reconciliation if data, stringData, and labels haven't changed
 			// This filters out metadata-only updates (ResourceVersion, ManagedFields, etc.)
 			if reflect.DeepEqual(oldSecret.Data, newSecret.Data) &&
-				reflect.DeepEqual(oldSecret.StringData, newSecret.StringData) &&
-				reflect.DeepEqual(oldSecret.Labels, newSecret.Labels) {
+				reflect.DeepEqual(oldSecret.Labels, newSecret.Labels) &&
+				oldSecret.Type == newSecret.Type {
 				return
 			}
 		}
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// For Secrets, check if only metadata changed (e.g., ResourceVersion updates from ESO)
// This prevents unnecessary reconciliations when external-secrets-operator updates metadata
if oldSecret, ok := evt.ObjectOld.(*corev1.Secret); ok {
if newSecret, ok := evt.ObjectNew.(*corev1.Secret); ok {
// Skip reconciliation if data, stringData, and labels haven't changed
// This filters out metadata-only updates (ResourceVersion, ManagedFields, etc.)
if reflect.DeepEqual(oldSecret.Data, newSecret.Data) &&
reflect.DeepEqual(oldSecret.StringData, newSecret.StringData) &&
reflect.DeepEqual(oldSecret.Labels, newSecret.Labels) {
return
}
}
}
// For Secrets, check if only metadata changed (e.g., ResourceVersion updates from ESO)
// This prevents unnecessary reconciliations when external-secrets-operator updates metadata
if oldSecret, ok := evt.ObjectOld.(*corev1.Secret); ok {
if newSecret, ok := evt.ObjectNew.(*corev1.Secret); ok {
// Skip reconciliation if data, stringData, and labels haven't changed
// This filters out metadata-only updates (ResourceVersion, ManagedFields, etc.)
if reflect.DeepEqual(oldSecret.Data, newSecret.Data) &&
reflect.DeepEqual(oldSecret.Labels, newSecret.Labels) &&
oldSecret.Type == newSecret.Type {
return
}
}
}
🤖 Prompt for AI Agents 
In internal/controller/dataprotectionapplication_controller.go around lines 197
to 210, the metadata-only filter misses changes to Secret.Type and includes an
ineffective StringData comparison; update the condition to also compare
oldSecret.Type != newSecret.Type (so type changes trigger reconciliation) and
remove (or stop relying on) the StringData check because Kubernetes returns
empty StringData on reads—keep the reflect.DeepEqual checks for Data and Labels
and return only when Type, Data, and Labels are equal.


q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
Name: dpaname,
Namespace: namespace,
Expand Down
Loading