Merge pull request #352 from thomas-dkmt/doc-high-availability

Complete documentation of High availability + add HA troubleshooting section

Author: stormi

docs/assets/img/xo-ha-enabled-disabled.png

@@ -1,106 +1,216 @@
 # High availability
 
-High availability (or HA) in XCP-ng world is the ability to detect a failed host and automatically boot all the VMs that were running on this host to the other alive machines.
+In XCP-ng, high availability (or HA) is the ability to detect a failed host and automatically boot all the VMs that were running on this host to the other alive machines.
 
 ## 📋 Introduction
 
-Implementing VM High availability (HA) is a real challenge: first because you need to reliably detect when a server has really failed to avoid unpredictable behavior. But that's not the only one. If you lose the network link but not the shared storage, how to ensure you will not write simultaneously on the storage and thus corrupt all your data?
+Implementing VM High availability (HA) is a real challenge.
+
+First, because you need to reliably detect when a server has really failed to avoid unpredictable behavior. 
+
+But that's not the only one. If you lose the network link but not the shared storage, how to ensure you will not write simultaneously on the storage and corrupt all your data as a result?
 
 We'll see how to protect your precious VM in multiple cases, and we'll illustrate that with real examples.
 
 :::info
-Even if you can have HA with only 2 hosts, it's strongly recommended to do it with at least 3 hosts, for obvious split-brains issues you might encounter.
+You can have high availability with as few as 2 hosts, but we strongly recommended to do it with 3 at the minimum, for obvious split-brain issues you might encounter.
 :::
 
 :::warning
-HA requires **far more maintenance** and will create some traps if you are not aware. In short, it comes at a cost. Before using it, **please carefully think about it**: do you **REALLY** need it? We saw people having less uptime because of HA than without. Because you **must understand** what you are doing every time you are rebooting or updating a host.
+High availability requires **far more maintenance** and will create some traps if you are not aware. In short, it comes at a cost. 
+
+Before using it, **please think about it carefully**: do you **REALLY** need it? We've seen people having less uptime using HA than when not using it, because you **must understand** what you are doing every time you reboot or update a host.
 :::
 
 ## 🎓 Concepts
 
 The pool concept allows hosts to exchange their data and status:
 
-* if you lose a host, it will be detected by the pool master.
-* if you lose the master, another host will take over the master role automatically.
+* If you lose a host, that will be detected by the pool master.
+* If you lose the master, another host will take over the master role automatically.
+
+To be sure a host is really unreachable, HA in XCP-ng uses multiple heartbeat mechanisms. As you saw in the introduction, it's not enough to check the network: what about storage? That's why there is also a specific heartbeat for shared storage between hosts in a pool. In fact, each host regularly writes some blocks in a dedicated VDI. That's the principle of the [Dead man's switch](http://en.wikipedia.org/wiki/Dead_man%27s_switch). 
+
+This concept is important, and it explains why you need to **configure high availability with a shared storage** (iSCSI, Fiber Channel or NFS) to avoid simultaneous writing in a VM disk.
+
+Here are the possible cases and how they are dealt with:
+
+* **Lost both network and storage heartbeat**: the host is considered unreachable and the HA plan is started
+* **Lost storage but not network**: if the host can contact a majority of pool members, it can stay alive. Indeed, in this scenario, there is no harm for the data (can't write to the VM disks). If the host is alone (i.e can't contact any other host or less than the majority), it goes for a reboot procedure.
+* **Lost network but not storage (worst case!)**: the host considers itself as problematic and starts a reboot procedure (hard power off and restart). This fencing procedure guarantees the sanity of your data.
+
+## Requirements
+
+Enabling HA in XCP-ng requires thorough planning and validation of several prerequisites:
+
+- **Pool-Level HA only**: HA can only be configured at the pool level, not across different pools.
+- **Minimum of 3 hosts recommended**: While HA can function with just 2 XCP-ng servers in a pool, we recommend using **at least 3** to prevent issues such as a split-brain scenario. With only 2 hosts, they risk getting fenced if the connection between them is lost.
+- **Shared storage requirements**: You must have shared storage available, including at least one iSCSI, NFS, XOSTOR or Fiber Channel LUN with a minimum size of **356 MB for the heartbeat Storage Repository (SR)**. The HA mechanism creates two volumes on this SR:
+    - A **4 MB heartbeat volume** for monitoring host status.
+    - A **256 MB metadata volume** to store pool master information for master failover situations.
+- **Dedicated heartbeat SR optional**: While it's not necessary to dedicate a separate SR for the heartbeat, you can choose to do so. Alternatively, you can use the same SR that hosts your VMs.
+- **Unsupported storage for heartbeat**: Storage using SMB or iSCSI authenticated via CHAP **cannot be used as the heartbeat SR**.
+- **Static IP addresses**: Make sure that all hosts have static IP addresses to avoid disruptions from DHCP servers potentially reassigning IPs.
+- **Dedicated bonded interface recommended**: For optimal reliability, we recommend using a dedicated bonded interface for the HA management network.
+- **VM agility for HA protection**: For a VM to be protected by HA, it must meet certain agility requirements:
+    - The VM’s virtual disks must **reside on shared storage**, such as iSCSI, NFS, or Fibre Channel LUN, which is also necessary for the storage heartbeat.
+    - The VM must **support live migration**.
+    - The VM should **not have a local DVD drive connection configured**.
+    - The VM’s network interfaces should be on **pool-wide networks**.
+
+:::tip
+For enabling HA, we **strongly recommend** to use a bonded management interface for servers in the pool, and to configure multipathed storage for the heartbeat SR.
+:::
+
 
-To be sure a host is really unreachable, HA in XCP-ng uses multiple heartbeat mechanisms. As you saw in the introduction, just checking the network isn't enough: what about the storage? That's why there is also a specific heartbeat for shared storage between hosts in a pool. In fact, each host regularly write some blocks in a dedicated VDI. That's the principle of the [Dead man switch](http://en.wikipedia.org/wiki/Dead_man%27s_switch). This concept is important, and it explains why you need to **configure HA with a shared storage** (iSCSI, Fiber Channel or NFS) to avoid simultaneous writing in a VM disk.
+If you create VLANs and bonded interfaces via the CLI, they might not be active or properly connected, causing a VM to appear non-agile and, therefore, unprotected by HA. 
 
-Here is the possibles cases and their answers:
+Use the `pif-plug` command in the CLI to activate VLAN and bond PIFs, ensuring the VM becomes agile. 
+
+Additionally, the `xe diagnostic-vm-status` CLI command can help identify why a VM isn’t agile, allowing you to take corrective action as needed.
 
-* **lost both network and storage heartbeat**: host is considered unreachable, HA plan is started
-* **lost storage but not network**: if the host can contact majority of pool members, it can stay alive. Indeed, in this scenario, there is no harm for the data (can't write to the VM disks). If the host is alone, i.e can't contact any other host or less than the majority, it decides to go for a reboot procedure.
-* **lost network but not storage (worst case!)**: the host considers itself as problematic, and start a reboot procedure (hard poweroff and restart). This fencing procedure guarantee the sanity of your data.
 
 ## ⚙️ Configuration
 
 ### Prepare the pool
 
-You can check if your pool have HA enable or not. In Xen Orchestra, you'll have a small "cloud" icon in the Home/pool view for each pool with HA enabled.
+You can check if your pool has HA enabled or not. 
+
+* In Xen Orchestra, for each pool where HA has been enabled, go to the **Home → Pool** view and you'll see a small "cloud" icon with a green check.
+* In the **Pool → Advanced** tab, you'll see a **High Availability** switch that shows if HA is enabled or not:
+
+![](../assets/img/xo-ha-enabled-disabled.png)
+
+To enable HA, just toggle it on, which gives you a SR selector as Heartbeat SR. 
 
-You can enable it with this xe CLI command:
+You can also enable it with this xe CLI command:
 
 ```
 xe pool-ha-enable heartbeat-sr-uuids=<SR_UUID>
 ```
 
 :::tip
-Remember that you need to use a shared SR to enable HA.
+Remember that you need to use a shared storage repository to enable high availability.
 :::
 
+Once enabled, HA status will be displayed with the green toggle.
+
 ### Maximum host failure number
 
-How many host failures you can tolerate before running out of options? For 2 hosts in a Pool, the answer is pretty simple: **1** is the maximum number. Why? Well, after loosing one host, it will be impossible to ensure a HA policy of the last one also fails.
+How many host failures you can tolerate before running out of options? For 2 hosts in a Pool, the answer is pretty simple: **1** is the maximum number: after losing one host, it will be impossible to ensure a HA policy if the last one also fails.
 
-This value can be computed by XCP-ng, and in our example case:
+XCP-ng can calculate this value for you. In our sample case, it looks like this:
 
 ```
 xe pool-ha-compute-max-host-failures-to-tolerate
 1
 ```
 
-But it could be also **0**. Because even if you lose 1 host, is there not enough RAM to boot the HA VM on the last one? If not, you can't ensure their survival. If you want to set the number yourself, you can do it with this command:
+But it could be also **0**. Because, even if you lose 1 host, is there not enough RAM to boot the HA VM on the last one? If not, you can't ensure their survival. 
+
+If you want to set the number yourself, you can do it with this command:
 
 ```
 xe pool-param-set ha-host-failures-to-tolerate=1 uuid=<Pool_UUID>
 ```
 
-When you have more hosts failed equal to this number, a system alert is raised: you are in a **over-commitment** situation.
+If more hosts fail than this number, the system will raise an **over-commitment** alert.
 
 ### Configure a VM for HA
 
-This is pretty straightforward with Xen Orchestra. Go on your VM page, then edit the *Advanced* panel: you just have to tick the HA checkbox.
+#### VM High availability modes
+
+In XCP-ng, you can choose between 3 high availability modes: restart, best-effort, and disabled:
+
+- **Restart**: if a protected VM cannot be immediately restarted after a server failure, HA will attempt to restart the VM when additional capacity becomes available in the pool. However, there is no guarantee that this attempt will be successful.
+- **Best-Effort**: for VMs configured with best-effort, HA will try to restart them on another host if their original host goes offline.\
+This attempt will only occur after all VMs set to the "restart" mode have been successfully restarted. HA will make only one attempt to restart a best-effort VM. If it fails, no further attempts will be made.
+- **Disabled**: if an unprotected VM or its host is stopped, HA will not attempt to restart the VM.
+
+#### Choosing a high availability mode
+
+This is pretty straightforward with Xen Orchestra. Go to the **Advanced** panel of your VM page and use the **HA** dropdown menu:
+
+![](../assets/img/xo-ha-selector.png)
 
 You can also do that configuration with *xe CLI*:
 
 ```
 xe vm-param-set uuid=<VM_UUID> ha-restart-priority=restart
 ```
 
+#### Start order
+
+##### What's the start order?
+
+The start order defines the sequence in which XCP-ng HA attempts to restart protected VMs following a failure. The order property of each protected VM determines this sequence.
+
+##### How and when does it apply?
+
+While the order property can be set for any VM, HA only uses it for VMs marked as **protected**. 
+
+The order value is an integer, with the default set to **0**, indicating the **highest priority**. VMs with an order value of 0 are restarted first, and those with higher values are restarted later in the sequence.
+
+##### How do I set the start order?
+
+You can set the order property value of a VM via the command-line interface:
+
+```
+xe vm-param-set uuid=<VM UUID> order=<number>
+```
+
+#### Configure HA timeout
+
+##### What's the HA timeout?
+
+The HA timeout represents the duration during which networking or storage might be inaccessible to the hosts in your pool. 
+
+If any XCP-ng server cannot regain access to networking or storage within the specified timeout period, it may self-fence and restart. 
+
+##### How do I configure it?
+
+The **default timeout is 60 seconds**, but you can adjust this value using the following command to suit your needs:
+
+```
+xe pool-param-set uuid=<pool UUID> other-config:default_ha_timeout=<timeout in seconds>
+```
+
 ## 🔧 Updates/maintenance
 
-Before any update or host maintenance, planned reboot and so on, you need to **ALWAYS** put your host in maintenance mode. If you don't do it, XAPI will consider it as a unplanned failure, and will act accordingly.
+Before any update or host maintenance, planned reboot and so on, **ALWAYS** put your host in maintenance mode. If you don't do that, XAPI will think it's an unplanned failure, and will act accordingly.
 
-If you have enough memory to put one host in maintenance (migrating all its VMs to other member of the pool), that will be alright. If you don't, you'll need to shutdown VMs manually **from a XAPI client** (Xen Orchestra or `xe`), and **NOT from inside the operating system**.
+If you have enough memory to put one host in maintenance (migrating all its VMs to another member of the pool), that will be alright. If you don't, you'll need to shut VMs down manually **from a XAPI client** (Xen Orchestra or `xe`), and **NOT from inside the operating system**.
 
 :::warning
-You **must be very careful before ANY maintenance task**, otherwise HA will kick in and provide unpleasant surprises. You have been warned.
+**Be very careful before doing ANY maintenance task**, otherwise HA will kick in and provide unpleasant surprises. You have been warned.
 :::
 
 ## ↔️ Behavior
 
 ### Halting the VM
 
-If you decide to shutdown the VM with `Xen Orchestra` or `xe`, the VM will be stopped normally, because XCP-ng knows that's what you want.
+If you shut the VM down with `Xen Orchestra` or `xe`, the VM will be stopped normally, because XCP-ng knows that's what you want.
+
+However, if you halt the VM directly in the guest OS (via the console or in SSH), XCP-ng is NOT aware of what's going on. The system will think the VM is down and will consider that an anomaly. As a result, the VM will be **started automatically!**. This behavior prevents an operator from shutting down the system and leaving the VM unavailable for a long time.
 
-But if you halt it directly in the guest OS (via the console or in SSH), XCP-ng is NOT aware of what's going on. For the system, it seems the VM is down and that's an anomaly. So, the VM will be **started automatically!**. This behavior prevent an operator to shutdown the system and leave the VM unavailable for a long time.
+:::tip
+
+Starting with XAPI 25.16.0, VM restart behavior can be changed. To do this, run this command:
+
+```
+xe pool-param-set uuid=... ha-reboot-vm-on-internal-shutdown=false
+```
+As a result, VMs that are shut down internally or through the API will restart the exact same way.
+
+:::
 
 ### Host failure
 
-We'll see 3 different scenarios on the host, with an example on 2 hosts, **lab1** and **lab2**:
+We'll see 3 different scenarios for the host, with an example on 2 hosts, **lab1** and **lab2**:
 
-* physically "hard" shutdown the server
-* physically remove the storage connection
-* physically remove the network connection
+* Physically power off the server.
+* Physically remove the **storage** connection.
+* Physically remove the **network** connection.
 
 **lab1** is not the *Pool Master*, but the results would be the same (just longer to test because of time to the other host becoming the master itself).
 
@@ -110,13 +220,13 @@ After each test, **Minion 1** go back to **lab1** to start in the exact same con
 
 #### Pull the power plug
 
-Now, we will decide to pull the plug for my host **lab1**:  this is exactly where my VM currently runs. After some time (when XAPI detect and report the lost of the host, in general 2 minutes), we can see that **lab1** is reported as Halted. In the same time, the VM **Minion 1** is booted on the other host running, **lab 2**:
+Now, we will decide to pull the plug for my host **lab1**: this is exactly where my VM currently runs. After some time (when XAPI detects and reports that the host is lost, which usually takes 2 minutes), we can see that **lab1** is reported as **Halted**. In the same time, the VM **Minion 1** is booted on the other running host - **lab 2**:
 
-If you decide to re-plug the host **lab1**, the host will be back online, without any VM on it, which is normal.
+If you decide to re-plug the **lab1** host, it will be back online, without any VM on it, which is normal.
 
 #### Pull the storage cable
 
-Another scenario: this time, we will unplug the iSCSI/NFS link on **lab1**, despite **Minion 1** is running on it.
+Another scenario: this time, we will unplug the iSCSI/NFS link on **lab1**, even though **Minion 1** is running on it.
 
 So? **Minion 1** lost access to its disks and after some time, **lab1** saw it can't access the heartbeat disk. Fencing protection is activated! The machine is rebooted, and after that, any `xe CLI` command on this host will give you that message:
 
@@ -132,4 +242,4 @@ Immediatly after fencing, **Minion 1** will be booted on the other host.
 
 #### Pull the network cable
 
-Finally, the worst case: leaving the storage operational but "cut" the (management) network interface. Same procedure: unplug physically the cable, and wait... Because **lab1** can't contact any other host of the pool (in this case, **lab2**), it decides to start the fencing procedure. The result is exaclty the same as the previous test. It's gone for the pool master, displayed as "Halted" until we re-plug the cable.
+Finally, the worst case: keep the storage operational, but "cut" the (management) network interface. Same procedure: unplug the cable physically and wait... Because **lab1** cannot contact any other host in the pool (in this case, **lab2**), it starts the fencing procedure. The result is exactly the same as the previous test. It's gone for the pool master, displayed as **Halted** until we re-plug the cable.