kubevirt-dm: S3 Detailed Design (S3 Usage & API Spec)

[weshayutin]

Description:
Expand the design for how we store qcow2 files and metadata in the BSL. Much of it is already defined in #2037 although we do need to add checksums.

Subtasks:

DOD:

[shubham-pampattiwar]

S3 Storage Design Details

The design in PR #2037 already covers the core storage layout. This comment expands on the details for implementation.

Data Flow: Where Checkpoint Data Comes From

The checkpoint data originates from KubeVirt's backup process:

1. Our controller creates VirtualMachineBackup CR
   ↓
2. KubeVirt/libvirt creates checkpoint + writes qcow2 to temp PVC
   ↓
3. VMBackup.status contains: checkpoint ID, backup type (full/incremental)
   ↓
4. Our datamover pod copies qcow2 files from PVC → S3
   ↓
5. Our controller creates/updates index.json with metadata

What comes from KubeVirt:

• Checkpoint ID (from VirtualMachineBackup.status)
• Backup type: full vs incremental (from VMBackup status)
• The actual qcow2 files (written to temp PVC)

What we manage ourselves:

• The index.json metadata files in S3
• The referencedBy array (we add Velero backup names for GC tracking)
• The checkpointChain in per-backup manifests (ordered list for restore)

BSL Storage Prefix

The kubevirt datamover uses a dedicated prefix derived from the BSL configuration:

<bsl-prefix>-kubevirt-datamover/

Note: This may need to be outside the main BSL prefix (e.g., if prefix is data, we create data-kubevirt-datamover) since Velero only allows its own files under the prefix. This needs validation during implementation.

Directory Structure

<bsl-prefix>-kubevirt-datamover/
├── manifests/                              # Per-backup metadata (we create)
│   └── <velero-backup-name>/
│       ├── index.json                      # Backup-level manifest
│       └── <vm-name>.json                  # Per-VM manifest for this backup
└── checkpoints/                            # qcow2 data + VM-level index
    └── <namespace>/
        └── <vm-name>/
            ├── index.json                  # Per-VM checkpoint index (we create)
            └── <checkpoint-id>/            # Checkpoint directory
                └── <vmb-name>-<disk-name>.qcow2  # From KubeVirt via temp PVC

Metadata File Schemas

1. Per-VM Checkpoint Index (checkpoints///index.json)

This is the primary tracking file for a VM's backup history. We create and maintain this file.

{
  "vmName": "string",
  "namespace": "string",
  "checkpoints": [
    {
      "id": "string",                    // From VirtualMachineBackup.status
      "type": "full | incremental",    // From VirtualMachineBackup.status
      "parent": "string | null",         // Parent checkpoint ID (null for full)
      "timestamp": "ISO8601",            // When checkpoint was created
      "vmBackup": "string",              // VirtualMachineBackup CR name
      "files": ["string"],               // qcow2 filenames from temp PVC
      "pvcs": ["string"],                // Associated PVC names
      "referencedBy": ["string"]         // Velero backup names (we maintain this)
    }
  ]
}

Key Design Points:

• referencedBy array enables garbage collection - when a backup is deleted, remove it from this array; when array is empty, checkpoint can be deleted
• parent field defines the incremental chain for restore
• files array supports multi-disk VMs

2. Per-Backup Index (manifests/<backup-name>/index.json)

Lightweight manifest identifying the backup.

{
  "backupName": "string",
  "timestamp": "ISO8601"
}

3. Per-Backup-Per-VM Manifest (manifests/<backup-name>/<vm-name>.json)

Links a specific backup to the checkpoint chain needed for restore.

{
  "namespace": "string",
  "name": "string",
  "checkpointChain": ["string"]         // Ordered list: [full, incr1, incr2, ...]
}

Restore uses this to:

1. Read the checkpointChain array
2. Fetch each checkpoint's qcow2 files from checkpoints///
3. Apply them in order using qemu-img rebase

S3/Object Store API

We reuse Velero's https://github.com/vmware-tanzu/velero/blob/7442d20/pkg/plugin/velero/object_store.go which provides cloud-agnostic operations:

┌─────────────────────┬──────────────────────────────┬──────────────────────────────────┐
│      Operation      │      ObjectStore Method      │             Use Case             │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ Upload qcow2        │ PutObject(bucket, key, body) │ Store backup data from temp PVC  │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ Download qcow2      │ GetObject(bucket, key)       │ Restore data                     │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ Read/Write metadata │ PutObject / GetObject        │ Manage index.json files          │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ List checkpoints    │ ListObjects(bucket, prefix)  │ Enumerate checkpoint directories │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ Delete checkpoint   │ DeleteObject(bucket, key)    │ GC orphaned qcow2 files          │
├─────────────────────┼──────────────────────────────┼──────────────────────────────────┤
│ Check existence     │ ObjectExists(bucket, key)    │ Verify checkpoint availability   │
└─────────────────────┴──────────────────────────────┴──────────────────────────────────┘

This gives us S3, GCS, and Azure Blob support for free via existing Velero plugins.

Garbage Collection on Backup Deletion

When a Velero backup is deleted (via DeleteItemAction):

1. Read manifests/<backup-name>/ to get list of VMs
2. For each VM:

• Read checkpoints/<ns>/<vm>/index.json
• Remove backup name from referencedBy arrays
• If any checkpoint's referencedBy becomes empty:
  • Delete qcow2 files in checkpoints/<ns>/<vm>/<checkpoint-id>/
  • Remove checkpoint entry from index.json
• Write updated index.json

3. Delete manifests/<backup-name>/ directory

Open Items for Implementation

1. Checksum storage (relates to kubevirt-dm: Checksum for all xferred data #2060): Where to store checksums? Options:

• Separate .sha256 file per qcow2
• Add checksums field to checkpoint metadata in index.json
• Use S3 object metadata (Content-MD5 header)

2. Large file handling: qcow2 files may exceed S3 single-part upload limits

• Need multipart upload support for files > 5GB

3. BSL credential access: Datamover pod needs BSL credentials

• Mount from Secret referenced in BSL spec
• Use projected ServiceAccount token for cloud-native auth

4. Prefix validation: Confirm whether we can use -kubevirt-datamover or need a different approach

[weshayutin]

nice work here folks :)