OCPBUGS-33013: certsyncpod+installerpod: Swap secret/cm directories atomically #2009

tchap · 2025-09-09T15:41:28Z

Currently it can happen that cert-syncer replaces some of the secret/configmap files successfully and then fails. This can lead to problems when these are e.g. TLS cert/key files and the directory gets inconsistent. This may seem transient, but when cert-syncer is terminated in the middle, it can later fail to start as the whole kube-apiserver gets into a crash loop.

This introduces a new staticpod.SwapDirectoriesAtomic, which uses unix.Renameat2 with RENAME_EXCHANGE flag set.

openshift-ci-robot · 2025-09-10T12:56:08Z

@tchap: This pull request references Jira Issue OCPBUGS-33013, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wangke19

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Currently it can happen that cert-syncer replaces some of the secret/configmap files successfully and then fails. This can lead to problems when these are e.g. TLS cert/key files and the directory gets inconsistent. This may seem transient, but when cert-syncer is terminated in the middle, it can later fail to start as the whole kube-apiserver gets into a crash loop.

This introduces a new staticpod.SwapDirectoriesAtomic, which uses unix.Renameat2 with RENAME_EXCHANGE flag set.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-09-10T12:57:22Z

@tchap: This pull request references Jira Issue OCPBUGS-33013, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wangke19

In response to this:

Currently it can happen that cert-syncer replaces some of the secret/configmap files successfully and then fails. This can lead to problems when these are e.g. TLS cert/key files and the directory gets inconsistent. This may seem transient, but when cert-syncer is terminated in the middle, it can later fail to start as the whole kube-apiserver gets into a crash loop.

This introduces a new staticpod.SwapDirectoriesAtomic, which uses unix.Renameat2 with RENAME_EXCHANGE flag set. This should not be a problem as this call is supported since Linux 3.15 on all modern file systems.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-09-10T13:11:53Z

@tchap: This pull request references Jira Issue OCPBUGS-33013, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wangke19

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Currently it can happen that cert-syncer replaces some of the secret/configmap files successfully and then fails. This can lead to problems when these are e.g. TLS cert/key files and the directory gets inconsistent. This may seem transient, but when cert-syncer is terminated in the middle, it can later fail to start as the whole kube-apiserver gets into a crash loop.

This introduces a new staticpod.SwapDirectoriesAtomic, which uses unix.Renameat2 with RENAME_EXCHANGE flag set.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tchap · 2025-09-10T13:12:25Z

I actually have to make sure this can be merged as this is only supported on Linux 3.15 or later.

/hold

tchap · 2025-09-10T13:47:04Z

This patch should be OK for RHEL 8 or later based on https://access.redhat.com/articles/3078

The latest CI for OCP 4.21 actually uses RHEL 9.6.

tchap · 2025-09-11T12:35:32Z

The PR using this change in cluster-kube-apiserver-operator seems to be passing on CI, I deem this ready.

/unhold

p0lyn0mial · 2025-09-15T07:51:42Z

@tchap is there a must-gather from an incident i could take a look at ?

tchap · 2025-09-15T07:57:19Z

@p0lyn0mial I think that I consulted this one: https://access.redhat.com/support/cases/#/case/03849958/discussion?attachmentId=a096R00003JpGgMQAV

p0lyn0mial · 2025-09-15T08:08:24Z

@vrutkovs do you have time to take a look at this issue ?

I think that the issue might be real. I think the issue is when a two file cert is replaced. It can happen that the server picks up the update and notices the public/private key mismatch and crashes. Is there a way to repo this issue ?

pkg/operator/staticpod/certsyncpod/certsync_controller.go

tchap · 2025-10-15T11:19:31Z

pkg/operator/staticpod/installerpod/cmd.go

-	filePerms := os.FileMode(0600)
-	if strings.HasSuffix(fullFilename, ".sh") {
-		filePerms = 0755
-	}


Hmm, didn't notice this check for custom permission setting...

tchap · 2025-10-16T15:27:38Z

pkg/operator/staticpod/certsyncpod/certsync_controller_sync_linux_test.go

+					strings.HasSuffix(path, "/staging/cert-sync/secrets") ||
+					strings.HasSuffix(path, "/staging/cert-sync/configmaps") ||
+					path == filepath.Join(controller.destinationDir, "configmaps") ||
+					path == filepath.Join(controller.destinationDir, "secrets") {


This is not too pretty, but meh.

p0lyn0mial

added a few more comments. overall lgtm.
please also test this pr with some operator e.g. kas-o

pkg/operator/staticpod/certsyncpod/certsync_controller.go

pkg/operator/staticpod/certsyncpod/certsync_controller_linux_test.go

pkg/operator/staticpod/installerpod/cmd.go

Use atomicdir.Sync to write target secret/configmap directories to be synchronized with the relevant objects. Added unit tests, but the coverage is not complete. Particularly filesystem operations failing are not being tested.

p0lyn0mial · 2025-11-04T15:53:56Z

lgtm

p0lyn0mial · 2025-11-04T15:55:00Z

/lgtm

openshift-ci · 2025-11-04T16:00:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: p0lyn0mial, tchap, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/operator/staticpod/OWNERS~~ [p0lyn0mial]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tchap · 2025-11-04T16:13:02Z

/unhold

let's merge this.

openshift-ci · 2025-11-04T16:36:05Z

@tchap: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-11-04T16:40:22Z

@tchap: Jira Issue OCPBUGS-33013: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-33013 has been moved to the MODIFIED state.

In response to this:

Currently it can happen that cert-syncer replaces some of the secret/configmap files successfully and then fails. This can lead to problems when these are e.g. TLS cert/key files and the directory gets inconsistent. This may seem transient, but when cert-syncer is terminated in the middle, it can later fail to start as the whole kube-apiserver gets into a crash loop.

This introduces a new staticpod.SwapDirectoriesAtomic, which uses unix.Renameat2 with RENAME_EXCHANGE flag set.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-11-13T14:40:18Z

Fix included in accepted release 4.21.0-0.nightly-2025-11-13-042845

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2025

openshift-ci bot requested review from p0lyn0mial and tkashem September 9, 2025 15:44

tchap force-pushed the atomic-certsync branch from b3eca57 to a389cbd Compare September 10, 2025 09:12

tchap mentioned this pull request Sep 10, 2025

OCPBUGS-33013: deps: Update library-go to update staticpod pkg openshift/cluster-kube-apiserver-operator#1917

Merged

tchap force-pushed the atomic-certsync branch 4 times, most recently from 8dec327 to 60f05a8 Compare September 10, 2025 12:45

tchap changed the title ~~WIP: certsyncpod: Swap secret/cm directories atomically~~ OCPBUGS-33013: certsyncpod: Swap secret/cm directories atomically Sep 10, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 10, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 10, 2025

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Sep 10, 2025

openshift-ci bot requested a review from wangke19 September 10, 2025 12:56

tchap changed the title ~~OCPBUGS-33013: certsyncpod: Swap secret/cm directories atomically~~ WIP: OCPBUGS-33013: certsyncpod: Swap secret/cm directories atomically Sep 10, 2025

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 10, 2025

tchap changed the title ~~WIP: OCPBUGS-33013: certsyncpod: Swap secret/cm directories atomically~~ OCPBUGS-33013: certsyncpod: Swap secret/cm directories atomically Sep 10, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 10, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 10, 2025

tchap force-pushed the atomic-certsync branch from 60f05a8 to f6df27a Compare September 10, 2025 14:03

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 11, 2025

p0lyn0mial reviewed Oct 15, 2025

View reviewed changes

pkg/operator/staticpod/certsyncpod/certsync_controller.go Outdated Show resolved Hide resolved

tchap commented Oct 15, 2025

View reviewed changes

tchap force-pushed the atomic-certsync branch from 6b52148 to 36195b3 Compare October 15, 2025 11:32

p0lyn0mial mentioned this pull request Oct 15, 2025

OCPBUGS-33013: atomicdir: Set permission for each file separately #2036

Merged

tchap force-pushed the atomic-certsync branch 2 times, most recently from f390998 to c11264b Compare October 16, 2025 15:26

tchap commented Oct 16, 2025

View reviewed changes

tchap force-pushed the atomic-certsync branch 2 times, most recently from deedcd7 to 542e674 Compare October 16, 2025 15:35

openshift-ci-robot mentioned this pull request Oct 20, 2025

OCPBUGS-33013: atomicdir: Split testing utils into a helper package #2038

Merged

tchap force-pushed the atomic-certsync branch from 542e674 to ba412ca Compare October 20, 2025 11:16

p0lyn0mial reviewed Oct 21, 2025

View reviewed changes

pkg/operator/staticpod/installerpod/cmd.go Outdated Show resolved Hide resolved

tchap force-pushed the atomic-certsync branch 2 times, most recently from 439e1f3 to 46d0eac Compare October 23, 2025 09:34

certsync+installer: Write object dirs atomically

f2df141

Use atomicdir.Sync to write target secret/configmap directories to be synchronized with the relevant objects. Added unit tests, but the coverage is not complete. Particularly filesystem operations failing are not being tested.

tchap force-pushed the atomic-certsync branch from 46d0eac to f2df141 Compare October 24, 2025 12:57

p0lyn0mial added approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. labels Nov 4, 2025

openshift-ci bot assigned p0lyn0mial Nov 4, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 4, 2025

openshift-merge-bot bot merged commit e9c2485 into openshift:master Nov 4, 2025
4 checks passed

tchap deleted the atomic-certsync branch November 5, 2025 12:07

OCPBUGS-33013: certsyncpod+installerpod: Swap secret/cm directories atomically #2009

OCPBUGS-33013: certsyncpod+installerpod: Swap secret/cm directories atomically #2009

Uh oh!

Conversation

tchap commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 10, 2025

Uh oh!

openshift-ci-robot commented Sep 10, 2025

Uh oh!

openshift-ci-robot commented Sep 10, 2025

Uh oh!

tchap commented Sep 10, 2025

Uh oh!

tchap commented Sep 10, 2025

Uh oh!

tchap commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

p0lyn0mial commented Sep 15, 2025

Uh oh!

tchap commented Sep 15, 2025

Uh oh!

p0lyn0mial commented Sep 15, 2025

Uh oh!

Uh oh!

tchap Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

tchap Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

p0lyn0mial left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

p0lyn0mial commented Nov 4, 2025

Uh oh!

p0lyn0mial commented Nov 4, 2025

Uh oh!

openshift-ci bot commented Nov 4, 2025

Uh oh!

tchap commented Nov 4, 2025

Uh oh!

openshift-ci bot commented Nov 4, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 4, 2025

Uh oh!

openshift-merge-robot commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tchap commented Sep 9, 2025 •

edited

Loading

tchap commented Sep 11, 2025 •

edited

Loading