From 9b1a279a70d480d0a6cff0fa41a698ff836463a9 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Fri, 21 Nov 2025 15:54:11 +0100 Subject: [PATCH 1/8] K8SPXC-1332 Documented custom CA certificates for backups and restoress to S3 storage dquote> dquote> The PR also improves TLS section in docs modified: docs/TLS.md modified: docs/backups-restore-to-new-cluster.md modified: docs/backups-storage.md modified: docs/operator.md new file: docs/tls-cert-manager.md new file: docs/tls-disable.md new file: docs/tls-manual.md new file: docs/tls-update.md modified: mkdocs-base.yml --- docs/TLS.md | 345 ++----------------------- docs/backups-restore-to-new-cluster.md | 43 ++- docs/backups-storage.md | 142 +++++----- docs/operator.md | 26 +- docs/tls-cert-manager.md | 68 +++++ docs/tls-disable.md | 30 +++ docs/tls-manual.md | 142 ++++++++++ docs/tls-update.md | 185 +++++++++++++ mkdocs-base.yml | 14 +- 9 files changed, 579 insertions(+), 416 deletions(-) create mode 100644 docs/tls-cert-manager.md create mode 100644 docs/tls-disable.md create mode 100644 docs/tls-manual.md create mode 100644 docs/tls-update.md diff --git a/docs/TLS.md b/docs/TLS.md index a63e5132..e3340cf5 100644 --- a/docs/TLS.md +++ b/docs/TLS.md @@ -3,347 +3,34 @@ The Percona Operator for MySQL uses Transport Layer Security (TLS) cryptographic protocol for the following types of communication: - -* Internal - communication between Percona XtraDB Cluster instances, - * External - communication between the client application and ProxySQL. +* Internal - communication between Percona XtraDB Cluster instances. The internal certificate is also used as an authorization method. -TLS security can be configured in several ways: - -* The Operator generates long-term certificates automatically if there are no -certificate secrets available (default option, and requires you renew them manually), - -* The Operator can use a specifically installed *cert-manager*, which will -automatically generate and renew short-term TLS certificates, - -* Certificates can be generated manually. - -You can also use pre-generated certificates available in the -`deploy/ssl-secrets.yaml` file for test purposes, but we strongly recommend -avoiding their usage on any production system! - -The following subsections explain how to configure TLS security with the -Operator yourself, as well as how to temporarily disable it if needed. - -## Install and use the *cert-manager* - -### About the *cert-manager* - -A [cert-manager :octicons-link-external-16:](https://cert-manager.io/docs/) is a Kubernetes certificate -management controller which is widely used to automate the management and -issuance of TLS certificates. It is community-driven, and open source. - -When you have already installed *cert-manager* and deploy the operator, the -operator requests a certificate from the *cert-manager*. The *cert-manager* acts -as a self-signed issuer and generates certificates. The Percona Operator -self-signed issuer is local to the operator namespace. This self-signed issuer -is created because Percona XtraDB Cluster requires all certificates issued -by the same . - -Self-signed issuer allows you to deploy and use the Percona -Operator without creating a clusterissuer separately. - -### Installation of the *cert-manager* - -The steps to install the *cert-manager* are the following: - -* Create a namespace, - -* Disable resource validations on the cert-manager namespace, - -* Install the cert-manager. - -The following commands perform all the needed actions: - -``` {.bash data-prompt="$" } -$ kubectl create namespace cert-manager -$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true -$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v{{ certmanagerversion }}/cert-manager.yaml -``` - -After the installation, you can verify the *cert-manager* by running the following command: - -``` {.bash data-prompt="$" } -$ kubectl get pods -n cert-manager -``` - -The result should display the *cert-manager* and webhook active and running. - -## Generate certificates manually - -To generate certificates manually, follow these steps: - - -1. Provision a Certificate Authority (CA) to generate TLS certificates - -2. Generate a CA key and certificate file with the server details - -3. Create the server TLS certificates using the CA keys, certs, and server -details - -The set of commands generate certificates with the following attributes: - -* `Server-pem` - Certificate - -* `Server-key.pem` - the private key - -* `ca.pem` - Certificate Authority - -You should generate certificates twice: one set is for external communications, -and another set is for internal ones. A secret created for the external use must -be added to `cr.yaml/spec/sslSecretName`. A certificate generated for internal -communications must be added to the `cr.yaml/spec/sslInternalSecretName`. - -``` {.bash data-prompt="$" } -$ cat < ca.pem.old - $ kubectl get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old - $ kubectl get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old - ``` - -3. Combine new and current `ca.pem` into a `ca.pem.combined` file: - - ``` {.bash data-prompt="$" } - $ cat ca.pem ca.pem.old >> ca.pem.combined - ``` - -4. Create a new Secrets object with *old* TLS certificate (`tls.pem.old`) - and key (`tls.key.old`), but a *new combined* `ca.pem` - (`ca.pem.combined`): - - ``` {.bash data-prompt="$" } - $ kubectl delete secret/cluster1-ssl-internal - $ kubectl create secret generic cluster1-ssl-internal --from-file=tls.crt=tls.pem.old --from-file=tls.key=tls.key.old --from-file=ca.crt=ca.pem.combined --type=kubernetes.io/tls - ``` - -5. The cluster will go through a rolling reconciliation, but it will do it - without problems, as every node has old TLS certificate/key, and both new - and old CA certificates. - -6. If new TLS certificate and key weren’t generated on step 1, - do that now. - -7. Create a new Secrets object for the second time: use new TLS certificate - (`server.pem` in the example) and its key (`server-key.pem`), and again - the combined CA certificate (`ca.pem.combined`): - - ``` {.bash data-prompt="$" } - $ kubectl delete secret/cluster1-ssl-internal - $ kubectl create secret generic cluster1-ssl-internal --from-file=tls.crt=server.pem --from-file=tls.key=server-key.pem --from-file=ca.crt=ca.pem.combined --type=kubernetes.io/tls - ``` - -8. The cluster will go through a rolling reconciliation, but it will do it - without problems, as every node already has a new CA certificate (as a part - of the combined CA certificate), and can successfully allow joiners with new - TLS certificate to join. Joiner node also has a combined CA certificate, so - it can authenticate against older TLS certificate. - -9. Create a final Secrets object: use new TLS certificate (`server.pmm`) and - its key (`server-key.pem`), and just the new CA certificate (`ca.pem`): - - ``` {.bash data-prompt="$" } - $ kubectl delete secret/cluster1-ssl-internal - $ kubectl create secret generic cluster1-ssl-internal --from-file=tls.crt=server.pem --from-file=tls.key=server-key.pem --from-file=ca.crt=ca.pem --type=kubernetes.io/tls - ``` - -10. The cluster will go through a rolling reconciliation, but it will do it - without problems: the old CA certificate is removed, and every node is - already using new TLS certificate and no nodes rely on the old CA - certificate any more. - -### Update certificates with downtime - -If your certificates have been already expired (or if you continue to use the -Operator version prior to 1.9.0), you should move through the -*pause - update Secrets - unpause* route as follows. - -1. Pause the cluster [in a standard way](pause.md), and make - sure it has reached its paused state. - -2. If cert-manager is used, delete issuer - and TLS certificates: - - ``` {.bash data-prompt="$" } - $ { - kubectl delete issuer/cluster1-pxc-ca - kubectl delete certificate/cluster1-ssl certificate/cluster1-ssl-internal - } - ``` - -3. Delete Secrets to force the SSL reconciliation: - - ``` {.bash data-prompt="$" } - $ kubectl delete secret/cluster1-ssl secret/cluster1-ssl-internal - ``` - -4. Check certificates to make sure reconciliation have succeeded. - -5. Unpause the cluster [in a standard way](pause.md), and make - sure it has reached its running state. - -### Keep certificates after deleting the cluster - -In case of cluster deletion, objects, created for SSL (Secret, certificate, and -issuer) are not deleted by default. +## TLS Certificates -If the user wants the cleanup of objects created for SSL, there is a [finalizers.delete-ssl](operator.md#finalizers-delete-ssl) -option in `deploy/cr.yaml`: if this finalizer is set, the Operator will delete -Secret, certificate and issuer after the cluster deletion event. +You can configure TLS security in several ways. -## Run Percona XtraDB Cluster without TLS +* By default, the Operator **generates long-term certificates** automatically during the cluster creation if there are no certificate secrets available. If you need new certificates, you must renew them manually. -Omitting TLS is also possible, but we recommend that you run your cluster with -the TLS protocol enabled. +* The Operator can use a *cert-manager*, which will automatically **generate and renew short-term TLS certificates**. You must explicitly install cert-manager for this scenario. -To have TLS protocol disabled (e.g. for demonstration purposes) set the -`unsafeFlags.tls` key to `true` and set the `tls.enabled` key to `false` -in the `deploy/cr.yaml` file: + The *cert-manager* acts as a self-signed issuer and generates certificates allowing you to deploy and use the + Percona Operator without a separate certificate issuer. -```yaml -... -spec: - ... - unsafeFlags - tls: true - ... - tls: - enabled: false -``` +* You can generate TLS certificates manually or obtain them from some other issuer and provide to the Operator. -### Enabling or disabling TLS on a running cluster +**For testing purposes**, you can use pre-generated certificates available in the `deploy/ssl-secrets.yaml` file. But we strongly recommend +**to not use them on any production system**! -You can set `tls.enabled` Custom Resource option to `true` or `false` to enable or disable TLS. However, doing this on a running cluster results in downtime and has the following side effects. +## TLS configuration -When the cluster is already running and the user switches `tls.enabled` to `false`, the Operator [pauses the cluster](pause.md), waits until all Pods are deleted, sets `unsafeFlags.tls` Custom Resource option to `true`, deletes TLS secrets, and [unpauses the cluster](pause.md). +The following sections provide guidelines how to: -Similarly, when the user switches `tls.enabled` to `true`, the Operator [pauses the cluster](pause.md), waits until all Pods are deleted, sets `unsafeFlags.tls` Custom Resource option to `false`, and [unpauses the cluster](pause.md). +* [Configure TLS security with the Operator using cert-manager](tls-cert-manager.md) +* [Generate certificates manually](tls-manual.md) +* [Update certificates](tls-update.md) +* [Disable TLS temporarily](tls-disable.md) -!!! warning - Don't change `tls.enabled` Custom Resource option when the cluster is in the process of enabling or disabling TLS: changing its value will immediately unpause the cluster even though the process has not yet completed. diff --git a/docs/backups-restore-to-new-cluster.md b/docs/backups-restore-to-new-cluster.md index 7cbd685f..04aa7eb3 100644 --- a/docs/backups-restore-to-new-cluster.md +++ b/docs/backups-restore-to-new-cluster.md @@ -5,10 +5,6 @@ You can restore from a backup as follows: * [On the same cluster where you made a backup](backups-restore.md) * On a new cluster deployed in a different Kubernetes-based environment. -To restore a backup, you will use the special restore configuration file. The -example of such file is [deploy/backup/restore.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/backup/restore.yaml). The list of options that can be used in it can -be found in the [restore options reference](operator.md#perconaxtradbclusterrestore-custom-resource-options). - This document focuses on the restore on a new cluster deployed in a different Kubernetes environment. ??? admonition "For Operator version 1.17.0 and earlier" @@ -35,19 +31,19 @@ You can check available options in the [restore options reference](operator.md#p ## Restore from a full backup -1. Set appropriate keys in the [deploy/backup/restore.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/backup/restore.yaml) file. +Configure the `PerconaXtraDBClusterRestore` Custom Resource. Specify the following keys in the [deploy/backup/restore.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/backup/restore.yaml) file: - * set `spec.pxcCluster` key to the name of the target cluster to restore - the backup on, +* set `spec.pxcCluster` key to the name of the target cluster to restore the backup on, - * set `spec.backupSource` subsection to point on the appropriate PVC, or - cloud storage: - - === "PVC volume" +* configure the `spec.backupSource` subsection to point to the PVC or the cloud storage where the backup is stored. + + === "PVC volume" - The `storageName` key should contain the storage name (which - should be configured in the main CR), and the `destination` key - should be equal to the PVC Name: + The `spec.backupSource` subsection should include: + + * `storageName` - the storage name, which + should be configured in the main CR + * `destination` key should be equal to the PVC Name: ```yaml ... @@ -69,24 +65,27 @@ You can check available options in the [restore options reference](operator.md#p === "S3-compatible storage" - The `destination` key should have value composed of three parts: - the `s3://` prefix, the S3 [bucket :octicons-link-external-16:](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html), - and the backup name, which you have already found out using the - `kubectl get pxc-backup` command. Also you should add necessary - S3 configuration keys, [same](backups-storage.md) as those used - to configure S3-compatible storage for backups in the - `deploy/cr.yaml` file: + The `spec.backupSource` subsection should include: + + * a destination key. Take it from the output of the `kubectl get pxc-backup` command. The destination consists of the `s3://` prefix, the S3 bucket name + and the backup name. + * the necessary [storage configuration keys](backups-storage.md#configure-storage-for-backups), just like in the `deploy/cr.yaml` file of the source cluster. + * `verifyTLS` to verify the storage server TLS certificate + * the custom TLS configuration if you use it for backups. Refer to the [backups-storage.md#configure-tls-verification-with-custom-certificates-for-s3-storage] section for more information. ```yaml ... backupSource: + verifyTLS: true destination: s3://S3-BUCKET-NAME/BACKUP-NAME s3: bucket: S3-BUCKET-NAME credentialsSecret: my-cluster-name-backup-s3 region: us-west-2 endpointUrl: https://URL-OF-THE-S3-COMPATIBLE-STORAGE - ... + caBundle: #If you use custom TLS certificates for S3 storage + name: minio-ca-bundle + key: ca.crt ``` === "Azure Blob storage" diff --git a/docs/backups-storage.md b/docs/backups-storage.md index 7df40e25..76878909 100644 --- a/docs/backups-storage.md +++ b/docs/backups-storage.md @@ -4,60 +4,51 @@ You can configure storage for backups in the `backup.storages` subsection of the Custom Resource, using the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/cr.yaml) configuration file. -You should also create the [Kubernetes Secret :octicons-link-external-16:](https://kubernetes.io/docs/concepts/configuration/secret/) -object with credentials needed to access the storage. +Before configuring the storage, you need to create a [Kubernetes Secret :octicons-link-external-16:](https://kubernetes.io/docs/concepts/configuration/secret/) object that contains the credentials needed to access your storage. === "Amazon S3 or S3-compatible storage" - 1. To store backups on the Amazon S3, you need to create a Secret with - the following values: + To use Amazon S3 or S3-compatible storage for backups, create a Secret object with your access credentials. Use the [deploy/backup/backup-secret-s3.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/backup/backup-secret-s3.yaml) file as an example. You must specify the following information: - * the `metadata.name` key is the name which you will further use to refer - your Kubernetes Secret, - * the `data.AWS_ACCESS_KEY_ID` and `data.AWS_SECRET_ACCESS_KEY` keys are - base64-encoded credentials used to access the storage (obviously these - keys should contain proper values to make the access possible). - - Create the Secrets file with these base64-encoded keys following the - [deploy/backup/backup-secret-s3.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/backup/backup-secret-s3.yaml) - example: + * the `metadata.name` key is the name the Kubernetes secret which you will reference in the Custom Resource + * `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are base64-encoded keys to access S3 storage - ```yaml - apiVersion: v1 - kind: Secret - metadata: - name: my-cluster-name-backup-s3 - type: Opaque - data: - AWS_ACCESS_KEY_ID: UkVQTEFDRS1XSVRILUFXUy1BQ0NFU1MtS0VZ - AWS_SECRET_ACCESS_KEY: UkVQTEFDRS1XSVRILUFXUy1TRUNSRVQtS0VZ - ``` + Use the following command to encode the keys: - !!! note + === ":simple-linux: in Linux" - You can use the following command to get a base64-encoded string from a plain text one: + ```bash + echo -n 'plain-text-string' | base64 --wrap=0 + ``` - === "in Linux" + === ":simple-apple: in macOS" - ``` {.bash data-prompt="$" } - $ echo -n 'plain-text-string' | base64 --wrap=0 - ``` + ```bash + echo -n 'plain-text-string' | base64 + ``` - === "in macOS" + Here's the example configuration of the Secret file: - ``` {.bash data-prompt="$" } - $ echo -n 'plain-text-string' | base64 - ``` + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: my-cluster-name-backup-s3 + type: Opaque + data: + AWS_ACCESS_KEY_ID: + AWS_SECRET_ACCESS_KEY: + ``` - Once the editing is over, create the Kubernetes Secret object as follows: + 1. Create the Kubernetes Secret object with this file: ``` {.bash data-prompt="$" } - $ kubectl apply -f deploy/backup/backup-secret-s3.yaml + $ kubectl apply -f deploy/backup/backup-secret-s3.yaml -n ``` !!! note - In case the previous backup attempt fails (because of a temporary + If the previous backup attempt fails (because of a temporary networking problem, etc.) the backup job tries to delete the unsuccessful backup leftovers first, and then makes a retry. Therefore there will be no backup retry without [DELETE permissions to the objects in the bucket :octicons-link-external-16:](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-actions.html). @@ -65,33 +56,16 @@ object with credentials needed to access the storage. can cause a similar problem. - 2. Put the data needed to access the S3-compatible cloud into the - `backup.storages` subsection of the Custom Resource. - - * `storages..type` should be set to `s3` (substitute the - part with some arbitrary name you will later use to refer this - storage when making backups and restores). - - * `storages..s3.credentialsSecret` key should be set to the name - used to refer your Kubernetes Secret (`my-cluster-name-backup-s3` in - the last example). - - * `storages..s3.bucket` and `storages..s3.region` should - contain the S3 bucket and region. + 2. Configure the storage in the Custom Resource. Modify the [`deploy/cr.yaml`](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/cr.yaml) file and define the following information: - * if you use some S3-compatible storage instead of the original Amazon - S3, add the [endpointURL :octicons-link-external-16:](https://docs.min.io/docs/aws-cli-with-minio.html) - key in the `s3` subsection, which should point to the actual cloud - used for backups. This value is specific to the cloud provider. For - example, using [Google Cloud :octicons-link-external-16:](https://cloud.google.com) involves the - [following :octicons-link-external-16:](https://storage.googleapis.com) endpointUrl: + * `storages..type` - set to `s3`. Substitute the + part with some name you will later use to refer this + storage when making backups and restores. - ```yaml - endpointUrl: https://storage.googleapis.com - ``` + * `storages..s3.credentialsSecret` set to the name of the Secret you created previously - The options within the `storages..s3` subsection are further - explained in the [Operator Custom Resource options](operator.md#operator-backup-section). + * `storages..s3.bucket` - where the data will be stored + * `storages..s3.region` - location of the bucket Here is an example of the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-xtradb-cluster-operator/blob/v{{release}}/deploy/cr.yaml) configuration file which configures Amazon S3 storage for backups: @@ -107,8 +81,56 @@ object with credentials needed to access the storage. bucket: S3-BACKUP-BUCKET-NAME-HERE region: us-west-2 credentialsSecret: my-cluster-name-backup-s3 + caBundle: + name: minio-ca-bundle + key: tls.cert ... ``` + + !!! note "S3-compatible storage" + + If you use S3-compatible storage instead of Amazon S3, add the `endpointUrl` option in the `s3` subsection. This points to your storage service and is specific to your cloud provider. For example, for MinIO: + + ```yaml + endpointUrl: https://minio-service:9000 + ``` + + For more configuration options, see the [Operator Custom Resource options](operator.md#operator-backup-section). + + ## Configure TLS verification with custom certificates for S3 storage + + !!! note "Version added: 1.19.0" + + You can use your organization's custom TLS / SSL certificates and instruct the Operator to securely verify TLS communication with S3 storage. + + To configure TLS verification with custom certificates, do the following: + + 1. Create the Secret object that contains the TLS certificate to access the S3 storage, the certificate's private key and the CA certificate. + 2. Modify the S3 storage configuration in the Custom Resource and specify the following information: + + * `storages..s3.caBundle.name` is the name of the Secret object you created previously + * `storages..s3.caBundle.key` is the CA certificate. + + Here's the example configuration: + + ```yaml + ... + backup: + ... + storages: + s3-us-west: + type: s3 + s3: + bucket: S3-BACKUP-BUCKET-NAME-HERE + region: us-west-2 + credentialsSecret: my-cluster-name-backup-s3 + caBundle: + name: minio-ca-bundle + key: ca.crt + ... + ``` + + The Operator will use this configuration to securely verify TLS communication with S3 storage during backups and restores. === "Microsoft Azure Blob storage" diff --git a/docs/operator.md b/docs/operator.md index 728f134a..67a1299f 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -2387,7 +2387,7 @@ The timeout value in seconds, after which backup job will automatically fail. | ----------- | ---------- | | :material-numeric-1-box: int | `3600` | -### backup.startingDeadlineSeconds +### `backup.startingDeadlineSeconds` The maximum time in seconds for a backup to start. The Operator compares the timestamp of the backup object against the current time. If the backup is not started within the set time, the Operator automatically marks it as "failed". @@ -2464,6 +2464,22 @@ The endpoint URL of the S3-compatible storage to be used (not needed for the ori | ----------- | ---------- | | :material-code-string: string | | +### `backup.storages.STORAGE-NAME.s3.caBundle.name` + +The name of the Secret that stores custom TLS certificates for TLS communication with S3 storage. See [Configure TLS verification with custom certificates for S3 storage](backups-storage.md#configure-storage-for-backups) for more information. + +| Value type | Example | +| ----------- | ---------- | +| :material-code-string: string | `s3-ca-bundle-secret` | + +### `backup.storages.STORAGE-NAME.s3.caBundle.key` + +The custom CA certificate for TLS communication with S3 storage. See [Configure TLS verification with custom certificates for S3 storage](backups-storage.md#configure-storage-for-backups) for more information. + +| Value type | Example | +| ----------- | ---------- | +| :material-code-string: string | `ca.crt` | + ### `backup.storages.STORAGE-NAME.persistentVolumeClaim.type` The persistent volume claim storage type. @@ -2826,6 +2842,14 @@ configuration file. This Custom Resource contains the following options: | credentialsSecret| string | The Secret name for the backup | true | | endpointUrl | string | A valid endpoint URL | false | | region | string | The region corresponding to the S3 bucket | false | +| caBundle | subdoc | Configuration for custom self-issued TLS certificcates | false + +#### backupSource.s3.caBundle subsection + +| Key | Value type | Description | Required | +| ---------------- | ----------------- | ---------------------------------------------- | -------- | +| name | string | The name of the Secret object that stores custom TLS certificates | true | +| key | string | The custom CA certificate file used to sign TLS certificates | true | ### backupSource.azure subsection diff --git a/docs/tls-cert-manager.md b/docs/tls-cert-manager.md new file mode 100644 index 00000000..0cc6eae9 --- /dev/null +++ b/docs/tls-cert-manager.md @@ -0,0 +1,68 @@ + +# Install and use the cert-manager + +## About the cert-manager + +A [cert-manager :octicons-link-external-16:](https://cert-manager.io/docs/) is a Kubernetes certificate +management controller which is widely used to automate the management and +issuance of TLS certificates. It is community-driven, and open source. + +When you have already installed cert-manager, nothing else is needed: just deploy the Operator, and the Operator will request a certificate from the cert-manager. + +The *cert-manager* acts +as a self-signed issuer and generates certificates. Certificates are valid for 3 months. + +![image](assets/images/certificates.svg) + +The Percona Operator +self-signed issuer is local to the operator namespace. This self-signed issuer +is created because Percona XtraDB Cluster requires all certificates issued +by the same CA (Certificate authority). + +Self-signed issuer allows you to deploy and use the Percona +Operator without creating a cluster issuer separately. + + + +## Install the *cert-manager* + +The cert-manager requires its own namespace + +The steps to install the *cert-manager* are the following: + +1. Create the `cert-manager` namespace: + + ```bash + kubectl create namespace cert-manager + ``` + +2. Disable resource validations on the `cert-manager` namespace: + + ```bash + kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true + ``` + +3. Install the cert-manager: + + ```bash + kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v{{ certmanagerversion }}/cert-manager.yaml + ``` + +4. Verify the *cert-manager* by running the following command: + + ```bash + kubectl get pods -n cert-manager + ``` + + The result should display the *cert-manager* and webhook active and running. + + ??? example "Expected output" + + ```{.text .no-copy} + NAME READY STATUS RESTARTS AGE + cert-manager-69f748766f-6chvt 1/1 Running 0 65s + cert-manager-cainjector-7cf6557c49-l2cwt 1/1 Running 0 66s + cert-manager-webhook-58f4cff74d-th4pp 1/1 Running 0 65s + ``` + +Once you create the database with the Operator, it will automatically trigger the cert-manager to create certificates. Whenever you check certificates for expiration, you will find that they are valid and short-term. diff --git a/docs/tls-disable.md b/docs/tls-disable.md new file mode 100644 index 00000000..e6ce93c2 --- /dev/null +++ b/docs/tls-disable.md @@ -0,0 +1,30 @@ +# Deploy Percona XtraDB Cluster without TLS + +You can deploy your Percona XtraDB Cluster without TLS, although we strongly recommend that you enable TLS for any production environment. + +## Disable TLS during initial deployment + +To disable TLS at deployment (for example, for demonstration or testing), edit your `deploy/cr.yaml` file. Set the `unsafeFlags.tls` key to `true` and the `tls.enabled` key to `false`: + +```yaml +... +spec: + ... + unsafeFlags: + tls: true + ... + tls: + enabled: false +``` + +## Enable or disable TLS on a running cluster + +You can enable or disable TLS at any time by changing the `tls.enabled` Custom Resource option to `true` to enable TLS, or to `false` to disable TLS. Be aware that changing this setting on a running cluster causes downtime and can introduce disruptions. + +- If you set `tls.enabled` to `false`, the Operator will [pause the cluster](pause.md), wait for all Pods to be deleted, set the `unsafeFlags.tls` option to `true`, delete the TLS secrets, and then [unpause the cluster](pause.md). + +- If you set `tls.enabled` to `true`, the Operator will [pause the cluster](pause.md), wait for all Pods to be deleted, set the `unsafeFlags.tls` option to `false`, and then [unpause the cluster](pause.md). + +!!! warning + + Do not change the `tls.enabled` option while the cluster is in the process of enabling or disabling TLS. Changing this value mid-process will immediately unpause the cluster, even if the operation is not complete. diff --git a/docs/tls-manual.md b/docs/tls-manual.md new file mode 100644 index 00000000..75fae8df --- /dev/null +++ b/docs/tls-manual.md @@ -0,0 +1,142 @@ + +# Generate certificates manually + +You can generate TLS certificates manually instead of using the Operator's automatic certificate generation. This approach gives you full control over certificate properties and is useful for production environments with specific security requirements. + +## What you'll create + +When you follow the steps from this guide, you'll generate these certificate files: + +* `server.pem` - Server certificate for MongoDB nodes +* `server-key.pem` - Private key for the server certificate +* `client.pem` - Client certificate for external connections +* `client-key.pem` - Private key for the client certificate +* `ca.pem` - Certificate Authority certificate +* `ca-key.pem` - Certificate Authority private key + +## Certificate requirements + +You need to create **two sets** of certificates: + +1. **External certificates** - for client connections from outside the cluster. +2. **Internal certificates** - for internal communication between Percona XtraDB Cluster nodes. + +After creating the certificates, you'll create two Kubernetes Secrets and reference them in your cluster configuration. + +## Prerequisites + +Before you start, make sure you have: + +* `cfssl` and `cfssljson` tools installed on your system +* Your cluster name and namespace ready +* Access to your Kubernetes cluster + +## Procedure + +### Generate certificates + +Replace `cluster1` and `my-namespace` with your actual cluster name and namespace in the commands below. + +1. Set your cluster variables + + ``` {.bash data-prompt="$" } + $ CLUSTER_NAME=my-cluster-name + $ NAMESPACE=my-namespace + ``` + +2. Create the Certificate Authority (CA) + + This command creates a root Certificate Authority that will sign all your certificates: + + ``` {.bash data-prompt="$" } + $ cat < ca.pem.old + kubectl get secret/ps-cluster1-ssl -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old + kubectl get secret/ps-cluster1-ssl -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old + ``` + +3. Combine new and current `ca.pem` into a `ca.pem.combined` file: + + ```bash + cat ca.pem ca.pem.old >> ca.pem.combined + ``` + +4. Create a new Secrets object with the *old* TLS certificate (`tls.pem.old`) and key (`tls.key.old`), but a *new combined* `ca.pem` (`ca.pem.combined`): + + ``` bash + kubectl create secret generic ps-cluster1-ssl \ + --from-file=tls.crt=server.pem.old \ + --from-file=tls.key=server-key.pem.old \ + --from-file=ca.crt=ca.pem.combined \ + --type=kubernetes.io/tls -o yaml --dry-run=client | kubectl apply -f - + ``` + +5. The cluster will go through a rolling restart. This process will not cause issues, because every node has the old TLS certificate/key, and both new + and old CA certificates. + +6. Create a new Secrets object again. This time use a new TLS certificate (`server.pem` in the example) and a new TLS key (`server-key.pem`), and again the combined CA certificate (`ca.pem.combined`): + + ``` bash + kubectl create secret generic ps-cluster1-ssl \ + --from-file=tls.crt=server.pem \ + --from-file=tls.key=server-key.pem \ + --from-file=ca.crt=ca.pem.combined \ + --type=kubernetes.io/tls -o yaml --dry-run=client | kubectl apply -f - + ``` + +7. The cluster will go through a rolling restart. This process will not cause issues, because every node already has a new CA certificate (as a part + of the combined CA certificate), and can successfully allow joiners with new + TLS certificate to join. A joiner node also has a combined CA certificate, so + it can authenticate against older TLS certificate. + +8. Create a final Secrets object: use the new TLS certificate (`server.pmm`) and + its key (`server-key.pem`), and only the new CA certificate (`ca.pem`): + + ``` bash + kubectl create secret generic ps-cluster1-ssl \ + --from-file=tls.crt=server.pem \ + --from-file=tls.key=server-key.pem \ + --from-file=ca.crt=ca.pem \ + --type=kubernetes.io/tls -o yaml --dry-run=client | kubectl apply -f - + ``` + +9. The cluster will go through a rolling restart, but it will do it + without problems: the old CA certificate is removed, and every node is + already using new TLS certificate and no nodes rely on the old CA + certificate any more. + +### Update certificates with downtime + +If your certificates have been already expired (or if you continue to use the +Operator version prior to 1.9.0), you should move through the +*pause - update Secrets - unpause* route as follows. + +1. Pause the cluster [in a standard way](pause.md), and make + sure it has reached its paused state. + +2. If cert-manager is used, delete issuer + and TLS certificates: + + ``` {.bash data-prompt="$" } + $ { + kubectl delete issuer/cluster1-pxc-ca + kubectl delete certificate/cluster1-ssl certificate/cluster1-ssl-internal + } + ``` + +3. Delete Secrets to force the SSL reconciliation: + + ``` {.bash data-prompt="$" } + kubectl delete secret/cluster1-ssl secret/cluster1-ssl-internal + ``` + +4. Check certificates to make sure reconciliation have succeeded. + +5. Unpause the cluster [in a standard way](pause.md), and make + sure it has reached its running state. + +### Keep certificates after deleting the cluster + +In case of cluster deletion, objects, created for SSL (Secret, certificate, and +issuer) are not deleted by default. + +If the user wants the cleanup of objects created for SSL, there is a [finalizers.delete-ssl](operator.md#finalizers-delete-ssl) +option in `deploy/cr.yaml`: if this finalizer is set, the Operator will delete +Secret, certificate and issuer after the cluster deletion event. diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 4de3d77c..771bd935 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -44,6 +44,7 @@ theme: - search.highlight - navigation.top - content.tabs.link + - content.code.copy extra_css: - https://unicons.iconscout.com/release/v3.0.3/css/line.css @@ -148,9 +149,9 @@ plugins: enabled: true # Google Analytics configuration - analytics: - provider: google - property: G-J4J70BNH0G +analytics: + provider: google + property: G-J4J70BNH0G extra: # Used in main.html template and can't be externalized @@ -224,7 +225,12 @@ nav: - dr-replication.md - dr-failover.md - dr-restore.md - - "Transport Encryption (TLS/SSL)": TLS.md + - "Transport Encryption (TLS/SSL)": + - "About TLS / SSL security": TLS.md + - "Install and use the cert-manager": tls-cert-manager.md + - "Generate certificates manually": tls-manual.md + - "Update certificates": tls-update.md + - "Run Percona XtraDB Cluster without TLS": tls-disable.md - "Data at rest encryption": encryption.md - "Telemetry": telemetry.md From 9d7bc904ea5468373611488bd45fd401ceb873d1 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Fri, 21 Nov 2025 15:58:20 +0100 Subject: [PATCH 2/8] Update docs/tls-manual.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/tls-manual.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tls-manual.md b/docs/tls-manual.md index 75fae8df..5ae64b0f 100644 --- a/docs/tls-manual.md +++ b/docs/tls-manual.md @@ -7,7 +7,7 @@ You can generate TLS certificates manually instead of using the Operator's autom When you follow the steps from this guide, you'll generate these certificate files: -* `server.pem` - Server certificate for MongoDB nodes +* `server.pem` - Server certificate for Percona XtraDB Cluster nodes * `server-key.pem` - Private key for the server certificate * `client.pem` - Client certificate for external connections * `client-key.pem` - Private key for the client certificate From cf938a732a2461f02981ef766e0769c618700cea Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Fri, 21 Nov 2025 15:58:26 +0100 Subject: [PATCH 3/8] Update docs/operator.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operator.md b/docs/operator.md index 67a1299f..cdc61987 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -2842,7 +2842,7 @@ configuration file. This Custom Resource contains the following options: | credentialsSecret| string | The Secret name for the backup | true | | endpointUrl | string | A valid endpoint URL | false | | region | string | The region corresponding to the S3 bucket | false | -| caBundle | subdoc | Configuration for custom self-issued TLS certificcates | false +| caBundle | subdoc | Configuration for custom self-issued TLS certificates | false #### backupSource.s3.caBundle subsection From 5fd71e1bcfd94da0f216f025d848831fae70df91 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Fri, 21 Nov 2025 15:58:48 +0100 Subject: [PATCH 4/8] Update docs/backups-restore-to-new-cluster.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/backups-restore-to-new-cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/backups-restore-to-new-cluster.md b/docs/backups-restore-to-new-cluster.md index 04aa7eb3..72d34b02 100644 --- a/docs/backups-restore-to-new-cluster.md +++ b/docs/backups-restore-to-new-cluster.md @@ -71,7 +71,7 @@ Configure the `PerconaXtraDBClusterRestore` Custom Resource. Specify the followi and the backup name. * the necessary [storage configuration keys](backups-storage.md#configure-storage-for-backups), just like in the `deploy/cr.yaml` file of the source cluster. * `verifyTLS` to verify the storage server TLS certificate - * the custom TLS configuration if you use it for backups. Refer to the [backups-storage.md#configure-tls-verification-with-custom-certificates-for-s3-storage] section for more information. + * the custom TLS configuration if you use it for backups. Refer to the [Configure TLS verification with custom certificates for S3 storage](backups-storage.md#configure-tls-verification-with-custom-certificates-for-s3-storage) section for more information. ```yaml ... From a0e69474fa431990cf67ac54142c72230a8c0626 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 26 Nov 2025 17:07:35 +0100 Subject: [PATCH 5/8] K8SPXC-1494 Documented TLS certificate and CA certificate validity durations in operator and cert-manager documentation. Added rules and limitations for customizing certificate durations. modified: docs/operator.md modified: docs/tls-cert-manager.md --- docs/operator.md | 16 ++++++++++++++ docs/tls-cert-manager.md | 46 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 61 insertions(+), 1 deletion(-) diff --git a/docs/operator.md b/docs/operator.md index 9905db9c..ef97eef3 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -239,6 +239,22 @@ Enables or disables the [TLS encryption](TLS.md). If set to `false`, | ----------- | ---------- | | :material-toggle-switch-outline: boolean | `true` | +### `tls.certValidityDuration` + +Validity period for TLS certificates. Minimum required validity is 1 hour. Durations lower than 1 hour are rejected. Setting the duration to exactly 1 hour prevents the Operator from generating the correct certificate object. + +| Value type | Example | +| ----------- | ---------- | +| :material-code-string: string | `2160h` | + +### `tls.caValidityDuration` + +Validity period for CA certificate. Minimum accepted duration is 730 hours (approximately 30 days). + +| Value type | Example | +| ----------- | ---------- | +| :material-code-string: string | `26280h` | + ### `tls.SANs` Additional domains (SAN) to be added to the TLS certificate within the extended cert-manager configuration. diff --git a/docs/tls-cert-manager.md b/docs/tls-cert-manager.md index 0cc6eae9..d9cabf86 100644 --- a/docs/tls-cert-manager.md +++ b/docs/tls-cert-manager.md @@ -65,4 +65,48 @@ The steps to install the *cert-manager* are the following: cert-manager-webhook-58f4cff74d-th4pp 1/1 Running 0 65s ``` -Once you create the database with the Operator, it will automatically trigger the cert-manager to create certificates. Whenever you check certificates for expiration, you will find that they are valid and short-term. +At this point, you can move on to [deploying the Operator and your database cluster](kubectl.md). + +## Customize certificate duration for cert-manager + +When you deploy the cluster using the default configuration, the Operator triggers the cert-manager to create certificates +with default duration of 90 days. + +You can also customize the certificate duration. For example, to align certificate lifetimes with your organization’s security and compliance policies. + +### Rules and limitations + +Check the following rules and limitations for setting up the certificate duration: + +1. You can set the duration **only when you create a new cluster**. Updating it in a running cluster is not supported. +2. The TLS certificate duration is subject to the following requirements: + + - The minimum accepted value is 1 hour. Durations below 1 hour are rejected. + - Do **not** set the duration to exactly 1 hour; the Operator will fail to generate the correct certificate object if you do. + - By default, cert-manager starts the renewal process when a certificate has one-third of its lifetime remaining, ensuring renewal before expiration. For example, if a certificate is valid for 1 hour, renewal will begin after approximately 40 minutes. + +3. Minimum CA certificate duration is 730 hours (approximately 30 days) + +### Configuration + +To set the custom duration, specify the following options in the Custom Resource: + +* `.spec.tls.certValidityDuration` – validity period for TLS certificates +* `.spec.tls.caValidityDuration` – validity period for the Certificate Authority (CA) + +Here's the example configuration: + +```yaml + tls: + enabled: true + certValidityDuration: 2160h + caValidityDuration: 26280h +``` + +Create a new cluster with this configuration: + +```bash +kubectl -f deploy/cr.yaml -n +``` + +To verify the durations, you can [check certificates for expiration](tls-update.md#check-your-certificates-for-expiration) at any time. This ensures your certificates are valid and helps you plan for renewals before they expire. From bb5fd7782834432a07f43103c849310a37a863aa Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Thu, 27 Nov 2025 15:03:32 +0100 Subject: [PATCH 6/8] Fixed cluster name to cluter1 to align with the rest of the docs --- docs/tls-update.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/tls-update.md b/docs/tls-update.md index fb9e9463..3dc4c97c 100644 --- a/docs/tls-update.md +++ b/docs/tls-update.md @@ -17,7 +17,7 @@ If you [use cert-manager](tls-cert-manager.md): 1. Check the necessary secrets names (`cluster1-ssl` and `cluster1-ssl-internal` by default): - ``` {.bash data-prompt="$" } + ```bash kubectl get certificate ``` @@ -90,9 +90,9 @@ as follows. and the TLS certificate key (`tls.key.old`): ```bash - kubectl get secret/ps-cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.pem.old - kubectl get secret/ps-cluster1-ssl -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old - kubectl get secret/ps-cluster1-ssl -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old + kubectl get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.pem.old + kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old + kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old ``` 3. Combine new and current `ca.pem` into a `ca.pem.combined` file: @@ -104,7 +104,7 @@ as follows. 4. Create a new Secrets object with the *old* TLS certificate (`tls.pem.old`) and key (`tls.key.old`), but a *new combined* `ca.pem` (`ca.pem.combined`): ``` bash - kubectl create secret generic ps-cluster1-ssl \ + kubectl create secret generic cluster1-ssl \ --from-file=tls.crt=server.pem.old \ --from-file=tls.key=server-key.pem.old \ --from-file=ca.crt=ca.pem.combined \ @@ -117,7 +117,7 @@ as follows. 6. Create a new Secrets object again. This time use a new TLS certificate (`server.pem` in the example) and a new TLS key (`server-key.pem`), and again the combined CA certificate (`ca.pem.combined`): ``` bash - kubectl create secret generic ps-cluster1-ssl \ + kubectl create secret generic cluster1-ssl \ --from-file=tls.crt=server.pem \ --from-file=tls.key=server-key.pem \ --from-file=ca.crt=ca.pem.combined \ @@ -133,7 +133,7 @@ as follows. its key (`server-key.pem`), and only the new CA certificate (`ca.pem`): ``` bash - kubectl create secret generic ps-cluster1-ssl \ + kubectl create secret generic cluster1-ssl \ --from-file=tls.crt=server.pem \ --from-file=tls.key=server-key.pem \ --from-file=ca.crt=ca.pem \ From beabc7e1c67cb4489f19846f2c08d330adb57772 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Mon, 1 Dec 2025 14:28:22 +0100 Subject: [PATCH 7/8] Added a warning about exact 730 hr value for CA duration --- docs/operator.md | 2 +- docs/tls-cert-manager.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operator.md b/docs/operator.md index ef97eef3..74fe5df9 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -249,7 +249,7 @@ Validity period for TLS certificates. Minimum required validity is 1 hour. Durat ### `tls.caValidityDuration` -Validity period for CA certificate. Minimum accepted duration is 730 hours (approximately 30 days). +Validity period for CA certificate. Minimum accepted duration is 730 hours (approximately 30 days). Setting this value to exactly 730 hours prevents the Operator from generating the correct certificate object. Recommended value is to be greater than 730 hours. | Value type | Example | | ----------- | ---------- | diff --git a/docs/tls-cert-manager.md b/docs/tls-cert-manager.md index d9cabf86..0488a920 100644 --- a/docs/tls-cert-manager.md +++ b/docs/tls-cert-manager.md @@ -85,7 +85,7 @@ Check the following rules and limitations for setting up the certificate duratio - Do **not** set the duration to exactly 1 hour; the Operator will fail to generate the correct certificate object if you do. - By default, cert-manager starts the renewal process when a certificate has one-third of its lifetime remaining, ensuring renewal before expiration. For example, if a certificate is valid for 1 hour, renewal will begin after approximately 40 minutes. -3. Minimum CA certificate duration is 730 hours (approximately 30 days) +3. Minimum CA certificate duration is 730 hours (approximately 30 days). Do not set the duration to exactly 730 hours; the Operator will fail to generate the correct certificate object if you do. ### Configuration From bf448d0cdb38b7380e9dea447e2c7df75a10e63e Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Tue, 2 Dec 2025 16:42:34 +0100 Subject: [PATCH 8/8] Updated after the review --- docs/operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operator.md b/docs/operator.md index 74fe5df9..2b71651e 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -249,7 +249,7 @@ Validity period for TLS certificates. Minimum required validity is 1 hour. Durat ### `tls.caValidityDuration` -Validity period for CA certificate. Minimum accepted duration is 730 hours (approximately 30 days). Setting this value to exactly 730 hours prevents the Operator from generating the correct certificate object. Recommended value is to be greater than 730 hours. +Validity period for CA certificate. Minimum accepted duration is 730 hours (approximately 30 days). Setting this value to exactly 730 hours prevents the Operator from generating the correct certificate object. You must set this value greater than 730 hours. | Value type | Example | | ----------- | ---------- |