Skip to content

feat: Custom backup and restoration#794

Draft
alexarefev wants to merge 17 commits intomainfrom
feature/custom_restoration
Draft

feat: Custom backup and restoration#794
alexarefev wants to merge 17 commits intomainfrom
feature/custom_restoration

Conversation

@alexarefev
Copy link
Copy Markdown
Collaborator

@alexarefev alexarefev commented Mar 10, 2026

Description

  • Provide an ability to manage periodic ETCD backups
  • Provide an ability to restore ETCD from periodic backup

Solution

  • Change the backup procedure to create CronJob that will manage periodic backup

Test Cases

TestCase 1

Make sure the backup is working

Test Configuration:

  • Hardware: 4CPU/8GB
  • OS: Ubuntu 22.04
  • Inventory: AllInOne

Steps:

  1. Run backup with the following options

Results:

Before After
--- CronJob is running and working correctly

TestCase 2

Make sure the restoration is working

Test Configuration:

  • Hardware: 4CPU/8GB
  • OS: Ubuntu 22.04
  • Inventory: AllInOne

Steps:

  1. Run restoration with the following options

Results:

Before After
--- Restoration has been finished successfully

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • There is no breaking changes, or migration patch is provided
  • Integration CI passed
  • There is no merge conflicts

@alexarefev alexarefev added improvement New feature or request python Pull requests that update Python code labels Mar 13, 2026
@alexarefev alexarefev marked this pull request as ready for review March 14, 2026 07:15
* make_descriptor
* pack

### Periodic ETCD backups
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, to enable periodic backups, we need to run separate backup procedure. Problems here is that backup procedure goal originally (as I understand) is to actually backup data, not to install backup job which will perform backups in future. It seems strange to use backup to perform install. If we need clusters to be installed with disabled etcd fsync, and we recommend that backup job is enabled when fsync is disabled, it would be safer if we install backup job right in install procedure (where we disable fsync).

The provisioner/storageclass need not to be present immediately, it could be installed later, and once installed it can provision the volume for the job

restore_plan:
etcd:
image: registry.k8s.io/etcd:3.6.6-0
snapshot: /opt/local-path-provisioner/pvc-e3b0d6c5-495d-4887-90d9-000d6b3d4d00_kube-system_etcd-backup/etcd-snapshot-20260220_103000.db
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if some other provisioner is used, e.g. some network-attached like NFS. In this case there will be no directory on the host by default, since volume is not mounted. Do we expect users to mount the volume on their own and find the mount point?

Maybe we could use some additional "backup download" job, which we will run only during restore, which will mount backup volume and copy latest backup to some well-known node direcotry. Then kubemarine will take backup from this well-known location on this particular host

if cluster.procedure_inventory.get('restore_plan', {}).get('etcd', {}).get('snapshot', {}):
cluster.log.debug('The particular snapshot will be used')
path_to_snap = cluster.procedure_inventory.get('restore_plan', {}).get('etcd', {}).get('snapshot', {})
first_control_plane = cluster.nodes['control-plane'].get_first_member()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should not assume that backup will be present on first master. E.g. local path provisioner could create volume on another node. Maybe using "download backup" job (and checking on which node it run) would be better

@alexarefev alexarefev added feature and removed improvement New feature or request labels Mar 16, 2026
@alexarefev alexarefev marked this pull request as draft March 16, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants