🐛 (go/v4): Fix flaky metrics e2e tests when webhooks are scaffolded

[camilamacedo86]

Problem

When the default e2e suite is run against a project that scaffolds admission
webhooks, the "metrics" test can fail with errors such as:

Error from server (InternalError): Internal error occurred: failed calling webhook:
Post "https://...webhook-service:443/validate--v1-pod": dial tcp ...:443:
connect: connection refused

The failure happens when the test launches the curl-metrics pod to verify the
metrics endpoint.

Root Cause

The original test assumed that, once the controller pod reported Running, the
webhook server was also ready to admit new pods. In reality:

1. The controller pod reaches Running.
2. cert-manager is still issuing the serving certificate secret for the webhook.
3. The webhook server only starts once that secret is created and mounted.
4. The metrics e2e test creates the curl pod while the webhook server is still initialising.
5. The validating webhook rejects the pod because it cannot accept connections yet.

Checking the controller logs for "Serving metrics server" does not guarantee the webhook server is listening, so the race persisted.

Solution

Gate the curl pod creation on two readiness checks:

1. Wait for the controller pod to report the Ready condition.
2. When webhooks are scaffolded, wait for the webhook service to publish EndpointSlice addresses (the Kubernetes-recommended readiness signal).

Controller readiness:

By("ensuring the controller pod is ready")
verifyControllerPodReady := func(g Gomega) {
    cmd := exec.Command("kubectl", "get", "pod", controllerPodName, "-n", namespace,
        "-o", "jsonpath={.status.conditions[?(@.type=='Ready')].status}")
    output, err := utils.Run(cmd)
    g.Expect(err).NotTo(HaveOccurred())
    g.Expect(output).To(Equal("True"), "Controller pod not ready")
}
Eventually(verifyControllerPodReady, 3*time.Minute, time.Second).Should(Succeed())

Webhook readiness (injected only when the project wires webhooks):

verifyWebhookEndpointsReady := func(g Gomega) {
    cmd := exec.Command("kubectl", "get", "endpointslices.discovery.k8s.io", "-n", namespace,
        "-l", "kubernetes.io/service-name=%s",
        "-o", "jsonpath={range .items[*]}{range .endpoints[*]}{.addresses[*]}{end}{end}")
    output, err := utils.Run(cmd)
    g.Expect(err).NotTo(HaveOccurred(), "Webhook endpoints should exist")
    g.Expect(output).ShouldNot(BeEmpty(), "Webhook endpoints not yet ready")
}
Eventually(verifyWebhookEndpointsReady, 3*time.Minute, time.Second).Should(Succeed())

A new scaffold marker (+kubebuilder:scaffold:e2e-metrics-webhooks-readiness) adds the EndpointSlice check only for webhook-enabled projects, keeping the base template generic.

Why this works

• The Ready condition requires all containers, volumes (including the webhook
  cert secret), and readiness probes to succeed, so the controller and its webhook server have finished initialising.
• EndpointSlices provide the modern, recommended signal that a service has routable endpoints, avoiding deprecated Endpoints objects.
• The previous log-based verification that the metrics server is serving remains in place, and the curl pod still validates the metrics output once the checks pass.

Notes on Kubernetes 1.33+

Kubernetes 1.33+ may exhibit a brief delay before the metrics endpoint becomes
available when controller-runtime's WithAuthenticationAndAuthorization() is
used with self-signed certificates. The readiness sequence above absorbs that
startup delay, so the curl pod is only launched after Kubernetes itself signals
that both the controller pod and webhook service are ready.

Closes: #5138 #5137

[camilamacedo86]

Hi @mayuka-c

Could you help review this one?

[mayuka-c]

/lgtm

This looks good to me since the controller pod readiness guarantees that the webhook server is ready. The approach which I had taken directly checks for webhook readiness which has additional overhead of defining and maintaining markers so that the webhook readiness checks only runs on PROJECT which has webhooks configured.

Thank you so much @camilamacedo86 :)

[k8s-ci-robot]

@mayuka-c: changing LGTM is restricted to collaborators

In response to this:

/lgtm

This looks good to me since the controller pod readiness guarantees that the webhook server is ready. The approach which I had taken directly checks for webhook readiness which has additional overhead of defining and maintaining markers so that the webhook readiness checks only runs on PROJECT which has webhooks configured.

Thank you so much @camilamacedo86 :)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, mayuka-c

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [camilamacedo86]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[camilamacedo86]

@mayuka-c that is the code that we must to inject.
Without it we still risking the flakes

Why?

Without waiting for the webhook service to publish endpoints, the curl pod can hit the validating webhook while it’s still initializing, producing the same “connection refused” failure. The EndpointSlice check is what gives us a deterministic signal that the webhook server is actually accepting traffic before we launch the curl metrics pod.

[mayuka-c]

Yes this would be required. I mean the approach which was taken here (To have the marker to check for webhook readiness).

Sorry, if I caused some misunderstandings here :)

[camilamacedo86]

Not at all — you helped a lot!
You identified the issue, and your effort was absolutely not lost.
I’ve added you as a co-author of this PR to give you proper credit for your contribution. 🙌

[camilamacedo86]

Hi @mayuka-c

Sorry, I changed it to do more tests and forgot to clarify.
I tried many options. The final solution is now in place.
Please, feel free to check

[mayuka-c]

Hi @mayuka-c

Sorry, I changed it to do more tests and forgot to clarify. I tried many options. The final solution is now in place. Please, feel free to check

Yeah this looks perfect for me. Once this is merged, I will be closing this PR here.
Thanks :)