Application pods depending on listener get stuck in pending when GKE nodes restart

[jeffsweep]

Affected Stackable version

25.7

Current and expected behavior

When Google Kubernetes Engine restarts nodes, the Superset pod gets stuck in a pending state. It refuses to start due an issue with being unable to scale-up. Presumably this would extend to other pods from other operators.

I traced this back to the listener PersistentVolumeClaim. It seems to be bound to the node(s) that no longer exists in the GKE cluster due to that node being restarted/removed/whatever. Deleting the PVC causes the listener operator to re-sync, and then everything is good again. But it requires manual intervention, which is not a good situation.

So basically whenever GKE swaps out nodes due to upgrades or other infrastructure reasons, the listener and everything connected to it get stuck.

Possible solution

Update operator code to delete PVC when nodes list changes.

Additional context

No response

Environment

v1.33.5-gke.1080000

Would you like to work on fixing this bug?

no

[lfrancke]

I think this is relevant: stackabletech/issues#770

In short: We're on it, but I'll let @sbernauer comment better.

[jeffsweep]

Sounds good. In the meantime, I am thinking of adding a monitoring pod/job/something that deletes the PVC when nodes change. Does that make sense as a stopgap solution?

[sbernauer]

With SDP 25.11 our defaults will change, so users should not run into this using the defaults.

Do you need to use a NodePort or are you willing to use a LoadBalancer instead? The pinning is only done for NodePorts.
In that case you can define this ListenerClass and use that instead:

listener-operator/deploy/helm/listener-operator/templates/listener-classes.yaml

Lines 18 to 23 in f7b4bf8

	apiVersion: listeners.stackable.tech/v1alpha1
	kind: ListenerClass
	metadata:
	name: external-stable
	spec:
	serviceType: LoadBalancer

As an alternative you can switch to the nightly listener-operator 0.0.0-dev with the feature we just merged this morning: #340. With this version you can configure if pinning should happen or not