OCPBUGS-74368: Stale operator metrics Service and ServiceMonitor resources need to be deleted

[djoshy]

- What I did
Added deletion manifests for the following resources:

• Service/machine-config-operator in openshift-machine-config-operator namespace
• ServiceMonitor/machine-config-operator in namespace openshift-machine-config-operator namespace

- How to verify it

• Create a cluster on 4.20.
• Verify that these resources exist.
• Then, upgrade to this PR(based on release-4.21 or main).
• These resources should be now deleted.

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-74368, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did
Added deletion manifests for the following resources:

• Service/machine-config-operator in openshift-machine-config-operator namespace
• ServiceMonitor/machine-config-operator in namespace openshift-machine-config-operator namespace

- How to verify it

• Create a cluster on 4.20.
• Verify that these resources exist.
• Then, upgrade to this PR(based on release-4.21 or main).
• These resources should be now deleted.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/cherry-pick release-4.21

[openshift-cherrypick-robot]

@djoshy: once the present PR merges, I will cherry-pick it on top of release-4.21 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@djoshy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-op-ocl	708b521	link	false	/test e2e-gcp-op-ocl

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[djoshy]

I verified this on a forced cluster upgrade from 4.20.12 to 4.22.0-0.nightly-2026-01-24-213011 (with an MCO image from this PR).

Install time cluster version history:

    history:
    - completionTime: "2026-01-26T14:48:25Z"
      image: registry.build11.ci.openshift.org/ci-ln-6778zgb/release@sha256:c9bae4933c711d664e15da5b98e6e057fda51697aef4f3ec8e932922aa969373
      startedTime: "2026-01-26T14:19:29Z"
      state: Completed
      verified: false
      version: 4.20.12

Resources:

$  oc get service
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
kube-rbac-proxy-crio        ClusterIP   172.30.203.234   <none>        9637/TCP              64m
machine-config-controller   ClusterIP   172.30.166.213   <none>        9001/TCP              64m
machine-config-daemon       ClusterIP   172.30.174.210   <none>        9001/TCP,8798/TCP     64m
machine-config-operator     ClusterIP   172.30.90.232    <none>        9001/TCP              64m
machine-config-server       ClusterIP   172.30.1.155     <none>        22623/TCP,22624/TCP   64m

$ oc get servicemonitor
NAME                        AGE
machine-config-controller   64m
machine-config-daemon       64m
machine-config-operator     64m

Force an upgrade to an OCP build based on the nightly above, with this PR:

$ oc adm upgrade --allow-explicit-upgrade --force --to-image quay.io/djoshy/ocp-release:cleanup-old
....
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image quay.io/djoshy/ocp-release:cleanup-old

Wait for upgrade to complete (oc get co should show all operators on the new build). Cluster version history will mark the upgrade as complete:

    history:
    - acceptedRisks: |-
        Target release version="" image="quay.io/djoshy/ocp-release:cleanup-old" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified
        Forced through blocking failures: Multiple precondition checks failed:
        * Precondition "ClusterVersionGiantHop" failed because of "MultipleMinorVersionsUpdate": 4.22.0-0.nightly-2026-01-24-213011 is more than one minor version beyond the current target 4.20.12 (4.22 > 4.(20+1)), and only updates within the current minor version or to the next minor version are supported.
        * Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=True (), so the update from 4.20.12 to 4.22.0-0.nightly-2026-01-24-213011 is probably neither recommended nor supported.
      completionTime: "2026-01-26T16:28:36Z"
      image: quay.io/djoshy/ocp-release:cleanup-old
      startedTime: "2026-01-26T15:17:18Z"
      state: Completed
      verified: false
      version: 4.22.0-0.nightly-2026-01-24-213011
    - completionTime: "2026-01-26T14:48:25Z"
      image: registry.build11.ci.openshift.org/ci-ln-6778zgb/release@sha256:c9bae4933c711d664e15da5b98e6e057fda51697aef4f3ec8e932922aa969373
      startedTime: "2026-01-26T14:19:29Z"
      state: Completed
      verified: false
      version: 4.20.12

It may take a minute or so after the upgrade is complete(on AWS at least), then verify resources in question:

$ oc get service
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
kube-rbac-proxy-crio        ClusterIP   172.30.203.234   <none>        9637/TCP              128m
machine-config-controller   ClusterIP   172.30.166.213   <none>        9001/TCP              128m
machine-config-daemon       ClusterIP   172.30.174.210   <none>        9001/TCP,8798/TCP     128m
machine-config-server       ClusterIP   172.30.1.155     <none>        22623/TCP,22624/TCP   128m

$ oc get servicemonitor
NAME                        AGE
machine-config-controller   128m
machine-config-daemon       128m

The operator metrics resources have been removed.

[ptalgulk01]

Pre-merge verified:

Verification Steps
Environment Setup
OpenShift Version: 4.22.0-0-2026-01-27-044238-test-ci-ln-ky0cd12-latest
Platform: AWS

Installed the cluster on this fix.
Able to see no openshift-machine-config-operator in service and servicemonitor in openshift-machine-config project

$ oc get service
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
kube-rbac-proxy-crio        ClusterIP   172.30.139.63   <none>        9637/TCP              47m
machine-config-controller   ClusterIP   172.30.68.63    <none>        9001/TCP              47m
machine-config-daemon       ClusterIP   172.30.247.18   <none>        9001/TCP,8798/TCP     47m
machine-config-server       ClusterIP   172.30.173.28   <none>        22623/TCP,22624/TCP   47m

$ oc get servicemonitor
NAME                        AGE
machine-config-controller   47m
machine-config-daemon       47m

Installed 4.20 cluster, check the below service and service monitor in openshift-machine-config-operator name

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.0-0.nightly-2026-01-26-202217   True        False         108m    Cluster version is 4.20.0-0.nightly-2026-01-26-202217

$ oc get servicemonitor
NAME                        AGE
machine-config-controller   3h2m
machine-config-daemon       3h2m
machine-config-operator     3h2m

$ oc get service
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
kube-rbac-proxy-crio        ClusterIP   172.30.235.30    <none>        9637/TCP              136m
machine-config-controller   ClusterIP   172.30.94.80     <none>        9001/TCP              136m
machine-config-daemon       ClusterIP   172.30.218.144   <none>        9001/TCP,8798/TCP     136m
machine-config-operator     ClusterIP   172.30.162.120   <none>        9001/TCP              136m
machine-config-server       ClusterIP   172.30.133.134   <none>        22623/TCP,22624/TCP   136m

Upgrade the cluster to this fix

$ oc get servicemonitor
NAME                        AGE
machine-config-controller   3h2m
machine-config-daemon       3h2m
machine-config-operator     3h2m

$  oc get service
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
kube-rbac-proxy-crio        ClusterIP   172.30.235.30    <none>        9637/TCP              3h28m
machine-config-controller   ClusterIP   172.30.94.80     <none>        9001/TCP              3h28m
machine-config-daemon       ClusterIP   172.30.218.144   <none>        9001/TCP,8798/TCP     3h28m
machine-config-server       ClusterIP   172.30.133.134   <none>        22623/TCP,22624/TCP   3h28m

No machine-config-operator metrics present.

/label qe-approved
/approved by @ptalgulk01

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-74368, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

- What I did
Added deletion manifests for the following resources:

• Service/machine-config-operator in openshift-machine-config-operator namespace
• ServiceMonitor/machine-config-operator in namespace openshift-machine-config-operator namespace

- How to verify it

• Create a cluster on 4.20.
• Verify that these resources exist.
• Then, upgrade to this PR(based on release-4.21 or main).
• These resources should be now deleted.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/verified by @ptalgulk01

[openshift-ci-robot]

@djoshy: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, isabella-janssen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [djoshy,isabella-janssen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD b092635 and 2 for PR HEAD 708b521 in total

[openshift-ci-robot]

@djoshy: Jira Issue Verification Checks: Jira Issue OCPBUGS-74368
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-74368 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

- What I did
Added deletion manifests for the following resources:

• Service/machine-config-operator in openshift-machine-config-operator namespace
• ServiceMonitor/machine-config-operator in namespace openshift-machine-config-operator namespace

- How to verify it

• Create a cluster on 4.20.
• Verify that these resources exist.
• Then, upgrade to this PR(based on release-4.21 or main).
• These resources should be now deleted.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@djoshy: new pull request created: #5600

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.