Alter system_auth and repair it before remote restores

[sync-by-unito]

Remote restores will require to keep the existing system_auth keyspace so that existing credentials can be retained. As token ownership will change, auth data could become unavailable after the restore. To avoid this and as preliminary step to a remote restore, the system_auth keyspace should be altered to be replicated on all nodes in the cluster and then repaired.

┆Issue is synchronized with this Jiraserver Feature by Unito
┆Epic: Remote Cluster Restore
┆Issue Number: K8SSAND-527
┆Priority: Medium

[jsanda]

@adejanovski I have some follow up questions/comments...

If the target cluster expands after the restore, should we continue updating system_auth so it replicates on all nodes? I assume it is not necessary, but I want to run it by you.

We do not yet have any sort of workflow in place within the operators for scheduling a repair. I think we could break this out into a separate ticket especially as it will involves reaper-operator. wdyt?

[adejanovski]

It's not necessary, and I would actually rather reduce the replication back to 3 post restore (but I'm not sure how we could achieve this).
For the repair, I was thinking of a simpler nodetool repair system_auth --full command on a single node which would do the trick as data is replicated on all nodes.

[sync-by-unito]

➤ Jeff DiNoto commented:

• Inspect cluster to learn how many nodes are present
  • Use management API to alter system_auth keyspace to replicate across all nodes
  • Use management API to trigger repair on system_auth