Skip to content

Containerized vllm and rhaiis#4

Merged
MiguelCarpio merged 2 commits intorhos-vaf:mainfrom
csibbitt:csibbitt_OSPRH-20192_containerized_vllm_and_rhaiis
Oct 10, 2025
Merged

Containerized vllm and rhaiis#4
MiguelCarpio merged 2 commits intorhos-vaf:mainfrom
csibbitt:csibbitt_OSPRH-20192_containerized_vllm_and_rhaiis

Conversation

@csibbitt
Copy link
Contributor

@csibbitt csibbitt commented Oct 8, 2025

  • Adds nvidia Container Toolkit
  • Configures Container Device Interface
  • Run vllm from a container
    • Frees us from the python version management
  • Updated flavor specs from testing
  • Also works with RHAIIS
  • BONUS: Idempotence checks for glance image and floating IP that got accidentally dropped on a previous refactor

* Adds nvidia Container Toolkit
* Configured Container Device Interface
* Run vllm from a container
  * Frees us from the python version management
* Updated flavor specs from testing
@csibbitt
Copy link
Contributor Author

csibbitt commented Oct 8, 2025

This PR moves the workload execution into a container. By default it runs upstream vllm / TinyLlama, but with the right options, it will run RHAIIS / Llama-2.3-1B instead.

I tested both workloads on my nested-passthrough setup, upstream vllm by applying the AnsibleTestCR.yaml as-is, and RHAIIS by adding the variables shown below. They are both working fine.

Variables for RHAIIS:

    gpu_validation_workload_container_image: "registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.0.0"
    gpu_validation_model_download_registry_username: "|3c5aa7e0..."
    gpu_validation_model_download_registry_password: "eyJhbGci..."
    gpu_validation_workload_cache_mount_path: "/opt/app-root/src/.cache"
    gpu_validation_model_name: RedHatAI/Llama-3.2-1B-Instruct-FP8
    gpu_validation_workload_userns: "--userns=keep-id:uid=1001"

Copy link
Contributor

@MiguelCarpio MiguelCarpio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@MiguelCarpio MiguelCarpio merged commit 2b59eb8 into rhos-vaf:main Oct 10, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants