Skip to content

Conversation

JaredforReal
Copy link
Contributor

@JaredforReal JaredforReal commented Sep 22, 2025

What type of PR is this?
feat: Implement testing profile

What this PR does / why we need it:
Implement a MVP of testing profile with mock vllm:

  • Added a configurable entrypoint for the router to allow overriding config via CONFIG_FILE(In Docker.extproc and 'scripts/entrypoint.sh`)
  • Implemented a mock vLLM service (FastAPI) with a testing profile in Compose ( In tools/mock_vllm)
  • Added a testing router config pointing to the mock vLLM(config/config.testing.yaml)

Which issue(s) this PR fixes:
Fixes #189

For reviewer:
This implementation, which has affected numerous files, may be worth discussing. Looking forward to any suggestions.

Copy link

github-actions bot commented Sep 22, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 config

Owners: @rootfs
Files changed:

  • config/config.testing.yaml

📁 Root Directory

Owners: @rootfs, @Xunzhuo
Files changed:

  • scripts/entrypoint.sh
  • Dockerfile.extproc
  • docker-compose.yml

📁 tools

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

  • tools/mock-vllm/Dockerfile
  • tools/mock-vllm/README.md
  • tools/mock-vllm/app.py
  • tools/mock-vllm/requirements.txt

📁 website

Owners: @Xunzhuo
Files changed:

  • website/docs/getting-started/docker-quickstart.md

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

Copy link

netlify bot commented Sep 22, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 19fd13a
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68d0a6ed7eb17800080267ce
😎 Deploy Preview https://deploy-preview-190--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copy link

netlify bot commented Sep 22, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 1bd0399
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68d28d585c6f6f0008aae424
😎 Deploy Preview https://deploy-preview-190--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>
@JaredforReal JaredforReal marked this pull request as ready for review September 22, 2025 08:13
Signed-off-by: JaredforReal <w13431838023@gmail.com>
@JaredforReal
Copy link
Contributor Author

@rootfs @Xunzhuo Thanks for your time! This PR is ready for review.

Signed-off-by: JaredforReal <w13431838023@gmail.com>
Copy link
Contributor

@tao12345666333 tao12345666333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, left one comment.

Signed-off-by: JaredforReal <w13431838023@gmail.com>
COPY src/semantic-router/go.mod src/semantic-router/go.sum src/semantic-router/
COPY candle-binding/go.mod candle-binding/semantic-router.go candle-binding/

# Pre-download Go modules to leverage Docker layer caching and fail fast if mirrors are unreachable
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move the dockerfile optimization to another PR?

@rootfs
Copy link
Collaborator

rootfs commented Sep 22, 2025

@JaredforReal lgtm, can you move the dockerfile optimization out this PR? we can review them in the followup one.

Copy link
Contributor

@yuluo-yx yuluo-yx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution.

There are currently 4 items related to docker. I'm wondering if we should move them to the "tools/deploy" or "tools/docker" directories? This would make the project root directory cleaner and neater.

@JaredforReal
Copy link
Contributor Author

@rootfs @yuluo-yx Thanks for your advice! I will work on them ASAP

@JaredforReal
Copy link
Contributor Author

Thanks for your contribution.

There are currently 4 items related to docker. I'm wondering if we should move them to the "tools/deploy" or "tools/docker" directories? This would make the project root directory cleaner and neater.

@yuluo-yx Got u, we already have a deploy/kubernetes in the root dir, which I haven't read and tried. I will try to wrap files related to Docker (including scripts/entrypoint.sh) in a tools/docker folder and make them more graceful and easier to understand. Thanks!!

@JaredforReal
Copy link
Contributor Author

JaredforReal commented Sep 23, 2025

@yuluo-yx Got u, we already have a deploy/kubernetes in the root dir, which I haven't read and tried. I will try to wrap files related to Docker (including scripts/entrypoint.sh) in a tools/docker folder and make them more graceful and easier to understand. Thanks!!

@yuluo-yx Thanks for your advice! I have tried to move docker files to tools/docker, and write another docker.mk for docker operations, like up, down, and testing to make it easier to use.
However, these changes affect the GitHub CI workflow, which makes this PR too complicated. I would like to do this in another PR with community's approval :)
So, I'm gonna leave the usage for now, and improve it later, Thanks again

Signed-off-by: JaredforReal <w13431838023@gmail.com>
@JaredforReal JaredforReal requested a review from rootfs September 23, 2025 06:37
@rootfs rootfs merged commit 34bb8fa into vllm-project:main Sep 23, 2025
10 of 11 checks passed
@JaredforReal JaredforReal deleted the compose branch September 24, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: implement a testing profile with mock vllm in docker compose
5 participants