Update cluster runtime upgrade with expand health check feature #3885
Conversation
The expand health check feature details are updated in learn document section concepts-cluster-upgrade-runtime.
|
@deepukraju : Thanks for your contribution! The author(s) and reviewer(s) have been notified to review your proposed change. |
|
Learn Build status updates of commit db537eb: ✅ Validation status: passed
For more details, please refer to the build report. |
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
Should we generalize the CCUVA term? May be say, "The upgrade can be continued when the customer executes the upgrade API."
There was a problem hiding this comment.
Do not use term CUVA, as customers do not know this.
State runtime upgrade or similar
There was a problem hiding this comment.
Updated the text to remove CUVA reference and used runtime upgrade.
| 3. **Comparison Process** - Comparison of current workloads with snapshot taken during start of upgrade. Report comparison status. | ||
| 4. **Health Check Handling** - On success proceed to next upgrade stage. For failure, based on inventory readiness check feature is enable or disable its handled as below. | ||
|
|
||
| | Upgrade Stage | UpgradeInventoryChecks Enable | UpgradeInventoryChecks Disable | |
There was a problem hiding this comment.
Can we clearly state the heading of the stage to say that this is the failure case? Something on the lines "Failure at Upgrade state". Trying to see if can put something that says what happens if there is a failure in this stage.
There was a problem hiding this comment.
The above statement says the table is for upgrade failure.
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
Do not use term CUVA, as customers do not know this.
State runtime upgrade or similar
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
we aren't really doing workload health checks. We are looking at the infrastructure of the tenant resources. I want to avoid indicating we check how their workloads are performing
There was a problem hiding this comment.
specifically right now we're checking for Nexus VM / Nexus AKS health. would it make sense to explicitly reference those for clarity?
There was a problem hiding this comment.
maybe something like "...triggered to conduct workload infrastructure availability" or "to conduct availability of the VM and NAKS health"
There was a problem hiding this comment.
Have mentioned tenant workload (Nexus Kubernetes Cluster and Virtual Machine). This will clarify we are referring to Nexus Kubernetes Cluster and Virtual Machine as tenant workload during health checks.
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
"When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage."
Is this information exposed to customers? How can it be viewed?
There was a problem hiding this comment.
I think logging is an internal mechanism that we probably don't want to inform customer about. I think in target-state we probably want to expose the information in the same form as we would if the feature were enabled (e.g. ARM properties, but TBD), but just don't block upgrades on it, and have it be informational only.
There was a problem hiding this comment.
What is the user experience look like for customers? I'm trying to understand how the customer would know the workloads aren't healthy after the runtime. I don't think we need to specify why but something to indicate the workload inventory check failed.
There was a problem hiding this comment.
The behavior is different for compute node-pool and other KCP and mgmt-plane servers. Its good to document this explicitly.
I will remove the log section.
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. | ||
|
|
||
| The Inventory Readiness Check feature performs workload health check after control-plane, management-plane and compute servers are upgraded during platform runtime upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify workload health state after different stages of platform runtime upgrade. the feature supports Nexus Kubernetes Cluster and Virtual Machine workloads. |
There was a problem hiding this comment.
There aren't workloads on the control plane. We need to be clear about what is being checked on these servers.
There was a problem hiding this comment.
the checks at each phase (KCP, NMP, Compute Rack 1, ....) aren't actually checking for workloads running on those node poolds, they are executing global checks as e.g. upgrading kubernetes could cause issues with workloads running on computes even though those compute machines haven't been upgraded yet. how much of this details do we want to include in the docs?
There was a problem hiding this comment.
Since we specify these are workload inventory checks, I would only specify the compute node scope. It may beneficial to reference the management node if you are checking CSNs.
There was a problem hiding this comment.
These are health check after different stages of upgrade. I have mentioned that.
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. | ||
|
|
||
| The Inventory Readiness Check feature performs workload health check after control-plane, management-plane and compute servers are upgraded during platform runtime upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify workload health state after different stages of platform runtime upgrade. the feature supports Nexus Kubernetes Cluster and Virtual Machine workloads. |
There was a problem hiding this comment.
What actions need to be performed by customer if the checks fail?
There was a problem hiding this comment.
I have provided the link for this.
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
How is the feature turned on/off?
There was a problem hiding this comment.
is there separate documentation produced for AFEC feature flags that we can link to?
There was a problem hiding this comment.
I would recommend to specify the functionality is feature flag enabled.
There was a problem hiding this comment.
I have mentioned functionality is feature flag enabled.
Need to provide a link to this.
|
#label:"aq-pr-triaged" |
prmerger-automator
bot
added
the
aq-pr-triaged
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. |
There was a problem hiding this comment.
inventory readiness check is triggered to conduct workload health checks
would be good to align on one term. i'm guessing "tenant workload health checks" may resonate the best based based on the rest of the nexus documentation? i think "inventory readiness check" is the name of the implementation which customer shouldn't need to know about.
There was a problem hiding this comment.
Change to tenant workload health checks
Updated the tenant workload health check process during cluster runtime upgrades, clarifying feature flag functionality and workflow steps.
|
Learn Build status updates of commit f592b78: ✅ Validation status: passed
For more details, please refer to the build report. |
Clarified behavior of health checks during runtime upgrade when feature is disabled.
|
Learn Build status updates of commit 1541c9b: ✅ Validation status: passed
For more details, please refer to the build report. |
|
|
||
| ## Nexus tenant workload health check during cluster runtime upgrade | ||
|
|
||
| During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the upgrade continues to next stage even after the health check failure. By default the feature is disabled. |
There was a problem hiding this comment.
do we want to reference the AFEC flag name here? (EnableUpgradeInventoryChecks) or are we relying on other documentation elsewhere for that?
| | Initial Snapshot | Upgrade failure | Upgrade continues to next stage | | ||
| | Control Plane Upgrade | Upgrade failure | Upgrade continues to next stage | | ||
| | Management Plane Upgrade | Upgrade failure | Upgrade continues to next stage | | ||
| | Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage | |
There was a problem hiding this comment.
| | Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage | | |
| | Compute server Upgrade | Upgrade paused, resumed when customer executes the continue API | Upgrade continues to next stage | |
The expand health check feature details are updated in learn document section concepts-cluster-upgrade-runtime.