Phase 0: default inference parameters — backend

[turisanapo]

Summary

Phase 0 (backend) of #425 — default inference parameters on model configuration.

• Add parameters field to ModelConfigSchema with temperature, top_p, max_tokens, frequency_penalty, presence_penalty, seed, stop, service_tier, and reasoning (including thinking translation for Anthropic Messages with display/summary/exclude support)
• Inject and translate defaults at the end of resolveModelAlias using ??= semantics — client values always win
• Add per-model catalog defaults via additionalProperties for key models (temperature, reasoning effort)
• 48 unit tests covering all parameters, client overrides, per-surface translation (Chat Completions, Messages, Responses), reasoning/thinking conversion, and no-op behavior

Resolution chain

request body  →  model config (branch.models[].parameters)  →  catalog model defaults

Parameter support

Parameter	Chat Completions	Messages	Responses
temperature	✅	✅	✅
top_p	✅	✅	✅
max_tokens	✅ max_completion_tokens	✅ max_tokens	✅ max_output_tokens
frequency_penalty	✅	✅	✅
presence_penalty	✅	✅	✅
seed	✅	✅	✅
stop	✅ stop	✅ stop_sequences (array)	—
service_tier	✅	✅	✅
reasoning	✅ reasoning + reasoning_effort	✅ → thinking (with display)	✅ reasoning + reasoning_effort

Test plan

• Unit tests for every parameter injection + client override
• Per-surface translation (max_tokens field names, stop → stop_sequences, reasoning → thinking)
• Reasoning display translation (summary → summarized/omitted, exclude → omitted, exclude precedence)
• Resolution precedence (client > model config > catalog defaults)
• No-op on empty params
• Typecheck clean (no new errors)

Closes #454

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Models now include preconfigured default parameters (temperature, reasoning effort) that enhance inference behavior.
  • Request parameters are intelligently applied with user values taking precedence over defaults.
  • Improved parameter handling across different API operations with proper field mapping and normalization.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6896aaeb-c1af-4a0d-95a0-7bc9ade9d26a

📥 Commits

Reviewing files that changed from the base of the PR and between 28ce36b and ccd21eb.

📒 Files selected for processing (4)

• apps/api/src/modules/providers/types.ts
• apps/gateway/src/gateway.ts
• apps/gateway/src/lib/hooks.test.ts
• apps/gateway/src/lib/hooks.ts

---

📝 Walkthrough

Walkthrough

This pull request introduces model parameter defaults configuration across the backend API and gateway. Changes include defining a ModelParametersSchema for inference parameters, wiring catalog-level defaults into model resolution, and implementing an injectModelParameters function that merges defaults with request-supplied values using operation-specific field translations.

Changes

Cohort / File(s)	Summary
Type Definitions apps/api/src/modules/providers/types.ts	Added ModelParametersSchema validating optional inference parameters (temperature, top_p, max_tokens, stop, reasoning, etc.), extended ModelConfigSchema with optional parameters field, and exported corresponding ModelParameters type.
Gateway Configuration apps/gateway/src/gateway.ts	Introduced MODEL_DEFAULTS keyed by model IDs and replaced tier-only metadata function with applyMeta that conditionally attaches model-specific temperature and reasoning effort defaults to catalog entries.
Request Processing Logic apps/gateway/src/lib/hooks.ts	Added readMeta(...) helper extracting metadata from model properties, implemented injectModelParameters(...) function applying parameter defaults to request body with operation-specific field translations (max_tokens remapping, stop sequence normalization, thinking object construction), and updated model resolution to merge and inject defaults.
Test Coverage apps/gateway/src/lib/hooks.test.ts	Added comprehensive test suite for injectModelParameters validating parameter merging semantics, operation-specific field mappings (chat vs messages vs responses), parameter translation rules (reasoning fields, stop sequences), and edge cases.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway as Gateway<br/>(hooks.ts)
    participant ModelCatalog as Model Catalog
    participant ModelConfig as Model Config<br/>(types.ts)
    
    Client->>Gateway: Request with body
    Gateway->>ModelCatalog: Resolve model alias
    ModelCatalog-->>Gateway: Model config + defaults
    
    Note over Gateway: readMeta() extracts<br/>defaults from additionalProperties
    
    Gateway->>ModelConfig: Get ModelParameters<br/>from config
    ModelConfig-->>Gateway: ModelParameters schema
    
    Note over Gateway: injectModelParameters()<br/>merges defaults with request body<br/>using ??= semantics
    
    Gateway->>Gateway: Translate parameters<br/>by operation type<br/>(chat/messages/responses)
    
    Note over Gateway: max_tokens → specific fields<br/>reasoning → reasoning_effort/thinking<br/>stop → stop_sequences
    
    Gateway-->>Client: Enhanced request body

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

• Phase 0: Default inference parameters — backend #454: Directly implements Phase 0 of default inference parameters backend with exact scope—ModelParametersSchema definition, injectModelParameters implementation with per-API-surface translation, per-model catalog defaults, and resolution chain with request-wins semantics.
• Support default inference parameters on model configuration #425: Parent epic for default inference parameters feature; these changes fulfill the backend Phase 0 requirements.
• #422: Related as it implements the same feature—adding ModelParameters schema to model configs and injecting defaults in the gateway.

Poem

🐰 With whiskers twitched and nose held high,
We've woven defaults through the sky!
Parameters dance in schemas bright,
Each API surface bathed in light,
Request takes throne, but defaults show the way,
Model defaults bloom in brilliant display! 🌱

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Phase 0: default inference parameters — backend' clearly identifies the main change: adding backend support for default inference parameters on model configurations.
Linked Issues check	✅ Passed	The PR implementation meets all coding requirements from issue #454: ModelParametersSchema with all required fields, injectModelParameters with ??= semantics, per-surface translations, catalog defaults support, and 48 unit tests.
Out of Scope Changes check	✅ Passed	All changes are directly related to the backend inference parameters implementation. No out-of-scope changes (UI, observability/telemetry) are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/default-inference-parameters

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 60 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[turisanapo]

@CodeRabbit please review

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@turisanapo Sure! I'll start reviewing the changes in PR #441 right away.

[review]

[heiwen]

I am a bit confused by the implemented approach. What's the purpose of tying defaults to a configured provider that cannot be changed at runtime? This should live on model aliases instead, should't it?

(minor: don't see a reason to include stop parameter)

[turisanapo]

I am a bit confused by the implemented approach. What's the purpose of tying defaults to a configured provider that cannot be changed at runtime? This should live on model aliases instead, should't it?

(minor: don't see a reason to include stop parameter)

Defaults are introduced at two levels:

1/ Model config: configurable per alias, resolved at runtime. These are "user defaults" and will be editable from the console in a different PR (Phase 1 from the issue design)

2/ Catalog model default: system-level. These provide out-of-the-box defaults for selected models.

The resolution chain is:

  request body  →  model config (alias)  →  catalog defaults
    (highest)        (user-configured)        (system)

Where the client always wins.

In this PR, I'm introducing the ModelParametersSchema, the injection/translation logic across all three API surfaces, and the resolution chain. The console UI for editing model config parameters comes in Phase 1.

Do we agree on having system-level defaults? Alternatively, we should drop them and only support user-configurable defaults.

[heiwen]

I don't see a requirement for catalogue / system level defaults right now. On the opposite, it can be quite confusing, since consumers will expect similar defaults as from other gateways / the upstream providers itself. Correct me if you have a clear customer requirement here.

One more setting to add is caching.

[heiwen]

Plus: to validate whether preset parameters can be overridden by request parameters in OpenRouter.

[turisanapo]

Plus: to validate whether preset parameters can be overridden by request parameters in OpenRouter.

This has already been explored. In OperRouter the client wins.

We have investigated other providers too, you can see more details in the parent issue: #425, Industry context section.

[heiwen]

Has the OpenRouter behavior been validated or is it just theoretical research?