Doc: Explain how to use a RayJob with Kueue and ProvisioningRequest despite's GKE single PodSet limitation

[fg91]

Description

The documentation explains how to run RayJobs with Kueue and queued provisioning on GKE. The documented manifests only work when the RayJob only has a head node but no workers. If one adds workers, GKE rejects the ProvisioningRequest because it only supports a single PodSet per request currently.

This PR documents how to circumvent this issue.

Related issues

Closes #59068 57839

Additional information

Created feature request to allow multiple podsets in GKE's issue tracker https://issuetracker.google.com/issues/452882313

[gemini-code-assist]

Code Review

This pull request adds important documentation for running RayJobs with Kueue on GKE, providing a workaround for GKE's single PodSet limitation. The addition of podSetMergePolicy and the explanatory note are good. My feedback aims to make the documentation more explicit about the requirement for identical head and worker node specifications to ensure the example is complete and works for users.

[gemini-code-assist]

The explanation of the workaround is helpful, but the instructions could be more explicit to ensure users can successfully follow the example. The note mentions that head and worker nodes need identical scheduling requirements, but the steps for deploying the RayJob don't guide the user on how to verify or configure this.

If the downloaded ray-job.pytorch-distributed-training.yaml doesn't have identical specs for the head and worker groups, the example will fail for users.

To improve clarity and ensure the example is robust, consider making the instructions more direct. For example, you could update the note like this:

:::{note}
Google Kubernetes Engine's queued provisioning feature currently supports only a single PodSet per request. To use a RayJob with both head and worker nodes, you must configure Kueue to merge the pods into a single PodSet.

This requires two things:
1.  Set `podSetMergePolicy: IdenticalWorkloadSchedulingRequirements` in the `ProvisioningRequestConfig`, as shown in this example.
2.  **Crucially, you must ensure your RayJob's head and worker group specs have identical resource requirements, affinities, and tolerations.** The example `ray-job.pytorch-distributed-training.yaml` must be configured this way for the job to be admitted by Kueue.
:::

This change would make the requirements clearer to the user.

[Future-Outlier]

LGTM, thank you!!

docs link: https://anyscale-ray--59070.com.readthedocs.build/en/59070/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html#kuberay-kueue-gang-scheduling-example