F #-: Improves llm validation and AI ready k8s documentation #466
Conversation
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
aleixrm
requested review from
cmoralopennebula,
imllorente,
prisorue and
rsmontero
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
mkomac
left a comment
mkomac
left a comment
There was a problem hiding this comment.
proposing to have same network name as on Scaleway deployment: admin_net
Hey @mkomac please, review the changes, this is already done, let me know if I forget to change it somewhere |
content/solutions/deployment_blueprints/ai-ready_opennebula/ai_ready_k8s.md
Outdated
Show resolved
Hide resolved
content/solutions/deployment_blueprints/ai-ready_opennebula/ai_ready_k8s.md
Show resolved
Hide resolved
content/solutions/deployment_blueprints/ai-ready_opennebula/ai_ready_k8s.md
Outdated
Show resolved
Hide resolved
content/solutions/deployment_blueprints/ai-ready_opennebula/ai_ready_k8s.md
Outdated
Show resolved
Hide resolved
content/solutions/deployment_blueprints/ai-ready_opennebula/ai_ready_k8s.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
…ia dynamo docs Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io>
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io> (cherry picked from commit da99372)
Signed-off-by: Aleix Ramírez <aramirez@opennebula.io> (cherry picked from commit da99372)
cmoralopennebula
left a comment
cmoralopennebula
left a comment
There was a problem hiding this comment.
Added some minor changes in the documentation for the Cloud Deployment step, and for the AI k8s ready.
| # sysctl -w net.ipv4.ip_forward=1 | ||
| # iptables -t nat -A POSTROUTING -s 192.168.100.0/24 -o enp129s0f0np0 -j MASQUERADE | ||
| # iptables-save | uniq | iptables-restore | ||
| netplan apply |
There was a problem hiding this comment.
There is no copy/paste button for this section here, it is being shown as an exit, not as an input that can be copy/pasted.
| ``` | ||
| To access the Kubernetes API from your localhost, use the kubeconfig file that is located in the `/etc/rancher/rke2/rke2.yaml` file of the workload cluster control plane node. Also, modify the local kubeconfig file for pointing to localhost, where `host_ip` is the ip address of the OpenNebula frontend IP address. and `control_plane_vm_ip` the workload cluster control plane VM IP address. | ||
| To access the Kubernetes API from your localhost, use the kubeconfig file that is located in the `/etc/rancher/rke2/rke2.yaml` file of the workload cluster control plane node. |
There was a problem hiding this comment.
This command is not showing the connection from your localhost, it is from the OpenNebula frontend host.
We have to either change the phrase to 'OpenNebula frontend host' or add an extra jump on the ssh to connect first to the frontend, then to router and finally control-plane.
| pods: 110 | ||
| ``` | ||
| 6. Finally, to use the PCI GPUs on the specific pod, add the `spec.runtimeClassName:nvidia` parameter in the pod/deploy manifest and set [`nvidia.com/gpu`](http://nvidia.com/gpu)`:1` as a requested resource. |
There was a problem hiding this comment.
There are no command to help nor support this part.
Can we add it?
cmoralopennebula
left a comment
cmoralopennebula
left a comment
There was a problem hiding this comment.
More messages and suggestions on the Dynamo implementation.
|
|
||
| ```shell | ||
| laptop$ cat <<EOF > storageClass.yaml | ||
| cat <<EOF > storageClass.yaml |
There was a problem hiding this comment.
Step 3 was not needed in my case.
The 'local-path' storage was already created from the previous step.
| EOF | ||
|
|
||
| laptop$ kubectl apply -f hf-secret.yaml | ||
| kubectl apply -f hf-secret.yaml |
There was a problem hiding this comment.
Should we use 'token' for the variable name, or 'hf-token-secret' as explained in the setp below?
| - -c | ||
| args: | ||
| - python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --is-prefill-worker 2>&1 | tee /tmp/vllm.log | ||
| EOF |
There was a problem hiding this comment.
Command kubectl -n dynamo-cloud get pods,svc does not appear as a command that can be copy/pasted.
It is included along the results that are generated from the command.
Line 319
|
|
||
| ```shell | ||
| laptop$ kubectl port-forward svc/<frontend_service> <local_port>:8000 & | ||
| kubectl port-forward svc/<frontend_service> <local_port>:8000 & |
There was a problem hiding this comment.
This command won't work, we need to specify the namespace as this was not deployed in the default one.
Change to
kubectl port-forward -n dynamo-cloud service/vllm-v1-disagg-router-frontend 9000:8000 &
| "stream": true, | ||
| "max_tokens": 300 | ||
| } | ||
| }' |
There was a problem hiding this comment.
Lines from 418 below are the output, we do not need copy/paste button on them.
Same happening on lines 374-393
4 participants
Description
Improves LLM validation and AI Ready kubernetes documentation with suggestions.
Branches to which this PR applies