validate applicaiton shows no problem when replication internal is not met

[abhijeet219]

Validate command output:

$ ./bin/odf dr validate application --name ceph1 --namespace openshift-dr-ops -o out
:star: Using config "config.yaml"
:star: Using report "out"
:mag_right: Validate config ...
   :white_check_mark: Config validated
:mag_right: Validate application ...
   :white_check_mark: Inspected application
   :white_check_mark: Gathered data from cluster "hub"
   :white_check_mark: Gathered data from cluster "sraghave-dc2"
   :white_check_mark: Gathered data from cluster "sraghave-dc1"
   :white_check_mark: Application validated
:white_check_mark: Validation completed (47 ok, 0 stale, 0 problem)

UI screenshot:

The drpc yaml of the application:

apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  annotations:
    drplacementcontrol.ramendr.openshift.io/app-namespace: openshift-dr-ops
    drplacementcontrol.ramendr.openshift.io/last-app-deployment-cluster: sraghave-dc1
  resourceVersion: '26329316'
  name: ceph1
  uid: b8126b9e-765e-478e-830e-7303f18ba47e
  creationTimestamp: '2025-10-10T18:17:15Z'
  generation: 6
  managedFields:
  namespace: openshift-dr-ops
  ownerReferences:
    - apiVersion: cluster.open-cluster-management.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Placement
      name: ceph1-placement-1
      uid: cfea9775-a6ce-4ca5-9979-412526c3258e
  finalizers:
    - drpc.ramendr.openshift.io/finalizer
  labels:
    cluster.open-cluster-management.io/backup: ramen
spec:
  action: Relocate
  drPolicyRef:
    apiVersion: ramendr.openshift.io/v1alpha1
    kind: DRPolicy
    name: odr-policy-5m
  failoverCluster: sraghave-dc2
  kubeObjectProtection:
    captureInterval: 5m0s
    recipeRef:
      name: ceph1
      namespace: ceph1
  placementRef:
    apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: ceph1-placement-1
    namespace: openshift-dr-ops
  preferredCluster: sraghave-dc1
  protectedNamespaces:
    - ceph1
  pvcSelector: {}
status:
  observedGeneration: 6
  resourceConditions:
    conditions:
      - lastTransitionTime: '2025-10-10T19:33:25Z'
        message: All VolSync PVCs are ready
        observedGeneration: 9
        reason: Ready
        status: 'True'
        type: DataReady
      - lastTransitionTime: '2025-10-10T19:41:53Z'
        message: All VolSync PVCs are protected
        observedGeneration: 9
        reason: DataProtected
        status: 'True'
        type: DataProtected
      - lastTransitionTime: '2025-10-23T11:59:15Z'
        message: All VolSync PVCs are protectedVRG object protectedKube objects protected
        observedGeneration: 9
        reason: DataProtected
        status: 'True'
        type: ClusterDataProtected
      - lastTransitionTime: '2025-10-10T19:33:26Z'
        message: No PVC conflict detected for VolumeSync schemeNo resource conflict
        observedGeneration: 9
        reason: NoConflictDetected
        status: 'True'
        type: NoClusterDataConflict
      - lastTransitionTime: '2025-10-10T19:33:24Z'
        message: 'PV and PVC restore skipped, an older generation as Primary has already applied the restore'
        observedGeneration: 9
        reason: Restored
        status: 'True'
        type: ClusterDataReady
      - lastTransitionTime: '2025-10-10T19:33:37Z'
        message: Kube objects restored
        observedGeneration: 9
        reason: KubeObjectsRestored
        status: 'True'
        type: KubeObjectsReady
    resourceMeta:
      generation: 9
      kind: VolumeReplicationGroup
      name: ceph1
      namespace: openshift-dr-ops
      protectedpvcs:
        - busybox-pvc-8
        - busybox-pvc-1
        - busybox-pvc-10
        - busybox-pvc-5
        - busybox-pvc-9
        - busybox-pvc-2
        - busybox-pvc-7
        - busybox-pvc-4
        - busybox-pvc-3
        - busybox-pvc-6
      pvcgroups:
        - grouped:
            - busybox-pvc-5
            - busybox-pvc-8
            - busybox-pvc-2
            - busybox-pvc-7
            - busybox-pvc-3
            - busybox-pvc-9
            - busybox-pvc-1
            - busybox-pvc-4
            - busybox-pvc-10
            - busybox-pvc-6
      resourceVersion: '53076525'
  lastKubeObjectProtectionTime: '2025-10-23T11:59:15Z'
  actionDuration: 8m33.810324312s
  lastUpdateTime: '2025-10-23T11:59:50Z'
  lastGroupSyncTime: '2025-10-23T06:16:06Z'
  lastGroupSyncDuration: 21m6.498890973s
  conditions:
    - lastTransitionTime: '2025-10-23T07:18:09Z'
      message: Completed
      observedGeneration: 6
      reason: Relocated
      status: 'True'
      type: Available
    - lastTransitionTime: '2025-10-10T19:33:53Z'
      message: Ready
      observedGeneration: 6
      reason: Success
      status: 'True'
      type: PeerReady
    - lastTransitionTime: '2025-10-23T11:59:20Z'
      message: VolumeReplicationGroup (openshift-dr-ops/ceph1) on cluster sraghave-dc1 is protecting required resources and data
      observedGeneration: 6
      reason: Protected
      status: 'True'
      type: Protected
  actionStartTime: '2025-10-10T19:25:20Z'
  phase: Relocated
  progression: Completed
  preferredDecision:
    clusterName: sraghave-dc1
    clusterNamespace: sraghave-dc1

[nirs]

The issue is we don't validate lastGroupSyncTime, only Protected=True. When adding this command, we assumed that the Protected condition reflect the protection status but turns out that it does not consider the replication updates.

To validate that replication interval is met, we need to compare lastGroupSyncTime with the current time in the ramen dr-cluster operator pod in the primary cluster. Since validation happens after the resource are gathered, and we want to support offline validation, the current time must be stored in the report to allow offline validation.

Ideally we should store the current pod time before or after collecting the VRG from the primary cluster, but gathering resources is done in kubectl-gather. What we have is the local time on the machine gathering the data when the VRG is written to storage. This time is not reliable since when creating a tarball with the files, the time may not be preserved.

We can handle this in kubectl-gather using an addon handling the VRG resource. When inspecting the VRG kubectl-gather can gather the current time from the ramen dr-cluster operator pod and store it in the addons/ directory. We may collect several vrgs depending on kubectl-gather configuration, but since all resources are gathered at the same time, we can keep single cluster time.

Knowing the cluster time seems useful regardless of VRG validation, so maybe it should be part of kubectl-gather core and not an addon. We can safely assume that the time in all nodes in the cluster is synchronized, so we don't need to get it from ramen dr-cluster operator pod, and instead can get it from the API server. We can get the time by performing HTTP Get on some resource with Cache-Control: no-cache and take the Date: HTTP header.

During validation we can use the current cluster time to validate lastGroupSyncTime.

The OpenShift console has undocumented logic to mark the application as critical if current time - lastGroupSyncTime > 3 * scheduling internal. Since we validated single point in time, and unmet replication interval is not expected, we should instead mark it as a problem if current time - lastGroupSyncTime > scheduling internal.

It would be better if ramen would report this information in the VRG using the Protected condition, or a new Replication condition. We will implement this in ramenctl only if adding this to ramen is not possible.