Extension for Default Environment Variables

[NaomiEisen]

Component

Setup/Standup

Desired use case or feature

The problem

Too many 'hidden' default values for successful model deployment.

Currently, the default deployed model is Facebook/opt-125m. For this small model, the existing default configuration (for example, the number of GPUs per decode pod) are just enough for successful deployment.
However, when switching to a larger model (such as Qwen/Qwen3-32B), the same default configuration will cause the stack deployment to fail.
I know it can be argued that adjusting these variables is the user’s responsibility... However, providing recommended (/default) configurations for larger models would save future developers significant time (the specific configurations are arguably 'trivial').

Proposed solution

We can take different paths for the solution- the main reason I created this issue is to discuss whether we want to introduce such a feature and how.
The most straightforward suggestion would be to create a few config files (extending env.sh) with recommended values for several model examples (ranging from small to large models).
Besides that, I think it would be beneficial to start breaking down the env.sh file into a more modular structure before it grows too extensively, using this as the first step toward that goal.

I would love to hear your take on this :)
Thanks!

Alternatives

No response

Additional context or screenshots

No response

[maugustosilva]

This is an interesting proposal, and I see multiple components as part of the solution.

1. Currently, and user that attempts to deploy a model which would not fit in the specified number of GPUs will be greet with a stern WARNING that it will inevitably fail, thanks to the "capacity planner" contributed by @jgchn. At this moment, the capacity planner operates in "warning only mode", but it will eventually be turned into "enforce mode" (and simply fail)

2. The "breaking down of the env.sh" is happening here [WiP] Declarative initial implementation #552, with @Vezio leading this implementation. Comments (and help) are welcome.

3. Even after 1) and 2) are completed, the idea of providing some example scenario files with parameters adequate for large models is good one and we should add it anyway.

[NaomiEisen]

This is an interesting proposal, and I see multiple components as part of the solution.

1. Currently, and user that attempts to deploy a model which would not fit in the specified number of GPUs will be greet with a stern WARNING that it will inevitably fail, thanks to the "capacity planner" contributed by @jgchn. At this moment, the capacity planner operates in "warning only mode", but it will eventually be turned into "enforce mode" (and simply fail)
2. The "breaking down of the env.sh" is happening here [WiP] Declarative initial implementation #552, with @Vezio leading this implementation. Comments (and help) are welcome.
3. Even after 1) and 2) are completed, the idea of providing some example scenario files with parameters adequate for large models is good one and we should add it anyway.

Thank you for your response :)
The "capacity planner" sounds awesome, maybe I missed the warning in my run, as it failed to deploy due to that exact problem (and unfortunately, more) - I'll double check that.
Good to hear about the Vezio's work as well!

Sounds like for the issue above, the immediate solution would be your third proposal, if it's welcomed anyways and would not intersect with the other work.