diff --git a/modules/migrate/pages/kubernetes/helm-to-operator.adoc b/modules/migrate/pages/kubernetes/helm-to-operator.adoc index 2d328e7cc6..f77b4ecae2 100644 --- a/modules/migrate/pages/kubernetes/helm-to-operator.adoc +++ b/modules/migrate/pages/kubernetes/helm-to-operator.adoc @@ -217,6 +217,84 @@ include::troubleshoot:partial$errors-and-solutions.adoc[tags=deployment] For more troubleshooting steps, see xref:manage:kubernetes/troubleshooting/k-troubleshoot.adoc[Troubleshoot Redpanda in Kubernetes]. +=== Resynchronize the bootstrap user password + +[CAUTION] +==== +The bootstrap user is designed to be set once at cluster creation and remain long-lived. Avoid regenerating the `-bootstrap-user` Secret when possible. Only use this procedure if the Secret has already been regenerated and you need to recover from the resulting authentication failure. +==== + +When the Redpanda Operator takes ownership of a SASL-enabled cluster, it manages a `-bootstrap-user` Secret that holds the superuser credentials. If this Secret is regenerated after migration (for example, if you delete a Helm-era Secret and expect the operator to recreate it with clean ownership), the operator writes a new random password into the Secret and the pod environment variables, but Redpanda's internal SCRAM database continues to hold the original password that the Helm chart set. On the next pod restart, `rpk` inside the pod reads the new password from its environment and fails to authenticate: + +[.no-copy] +---- +SASL_AUTHENTICATION_FAILED: Invalid credentials +---- + +The Redpanda Operator does not resynchronize this password for you: changing a SCRAM password requires authenticating with the old password, and the bootstrap Secret only tracks one credential at a time. You must update the SCRAM database manually, using the old password to authenticate and the new password from the regenerated Secret as the target. + +[TIP] +==== +If you have not regenerated the bootstrap-user Secret yet, back up the current password first so you have it available if you need to run this procedure later: + +[,bash] +---- +kubectl --namespace get secret -bootstrap-user \ + -o jsonpath='{.data.password}' | base64 -d +---- +==== + +[NOTE] +==== +Only perform these steps if `rpk` fails SASL authentication after migration. You also need the original superuser password that the Helm chart used. If you no longer have it, you may need to restore from backup or contact Redpanda support. +==== + +. Confirm that every broker pod exposes the same `RPK_PASS` value. The resync step in the next item updates the SCRAM database to whatever the pod's env currently holds, so every pod must agree on that value first: ++ +```bash +for pod in $(kubectl --namespace get pods -l app.kubernetes.io/component=redpanda-statefulset -o name); do + echo "$pod: $(kubectl --namespace exec $pod -c redpanda -- printenv RPK_PASS)" +done +``` ++ +-- +If every pod reports the same password, continue to the next step. + +If one or more pods report a different `RPK_PASS` value from the rest, the bootstrap Secret on those pods' nodes is still cached by the kubelet. The Redpanda Operator creates the `-bootstrap-user` Secret with `immutable: true`, and kubelet's local Secret cache does not always invalidate when an immutable Secret is deleted and re-created with the same name. To force affected pods to pick up the current Secret value: + +- Force-delete the pod: `kubectl --namespace delete pod --force --grace-period=0`. +- If the env still disagrees after the pod is re-created, drain the node that hosted the pod so the StatefulSet schedules it elsewhere: `kubectl drain --ignore-daemonsets --delete-emptydir-data`. + +Repeat the check until every pod reports the same value before continuing. +-- + +. Open a shell in any broker pod. The pod's environment already exposes the new password from the regenerated Secret: ++ +```bash +kubectl --namespace exec -it -0 -c redpanda -- bash +``` + +. Resynchronize the SCRAM database. Replace `` with the superuser password set by the Helm chart before migration: ++ +[,bash] +---- +export RPK_NEW_PASS="$RPK_PASS" <1> +export RPK_PASS="" <2> +rpk acl user update "$RPK_USER" --mechanism "$RPK_SASL_MECHANISM" --new-password "$RPK_NEW_PASS" <3> +export RPK_PASS="$RPK_NEW_PASS" <4> +rpk cluster info <5> +---- ++ +-- +<1> Save the new password that the operator wrote to the Secret. `$RPK_PASS` in the pod already points to this value. +<2> Switch `rpk` to authenticate with the original Helm-era password so the next command can reach the admin API. +<3> Update the superuser's password in Redpanda's SCRAM database to match the new password in the Secret. +<4> Restore `$RPK_PASS` to the new password for subsequent commands. +<5> Verify that authentication succeeds with the new password. +-- + +After the update, the SCRAM database and the bootstrap Secret agree on the password, and subsequent pod restarts continue to authenticate cleanly. + === Open an issue If you cannot solve the issue or need assistance during the migration process, https://github.com/redpanda-data/redpanda-operator/issues/new/choose[open a GitHub issue^]. Before opening a new issue, search the existing issues on GitHub to see if someone has already reported a similar problem or if any relevant discussions can help you.