feat(providers): /models probe for custom endpoint validation#34
Merged
DonPrus merged 16 commits intonullclaw:mainfrom May 5, 2026
Merged
feat(providers): /models probe for custom endpoint validation#34DonPrus merged 16 commits intonullclaw:mainfrom
DonPrus merged 16 commits intonullclaw:mainfrom
Conversation
- GET /api/providers/probe-models?base_url=...&api_key=... fetches available model IDs from any OpenAI-compatible /models endpoint - handleCreate, handleUpdate, handleValidate now probe custom providers via /models instead of silently skipping; create/update are non-blocking (always save, record validation result); validate returns live status - Frontend: Fetch Models button in add/edit forms for custom providers; clickable model chips populate the model field - 9 new backend tests (buildModelsUrl, parseModelIdsJson, isProbeModelsPath, handleProbeModels, handleValidate behaviour, handleCreate timestamp)
…base_url - Replace all real provider names/URLs/ports in test blocks with generic values (custom-llm, https://example.com/v1, sk-test-key, port 19999) - Fix State.load() silently dropping base_url on round-trip: the struct literal in the saved_providers loop was missing .base_url = owned_base_url - Remove dangling orphan load-block left by a partial test edit - UI, syncProviderToInstances, and /models probe changes from prior work included in this commit (were already staged but uncommitted)
- Add ui/src/lib/providers.ts as shared source of truth for provider constants (PROVIDER_OPTIONS, OPENAI_COMPATIBLE_VALUE, LOCAL_PROVIDERS, KNOWN_PROVIDER_VALUES, mergeWithManifestOptions) - providers/+page.svelte and ProviderList.svelte now import from the shared module; openai-compatible always visible in wizard dropdown - orchestrator: extractCustomProvider strips custom provider fields before --from-json call so the binary sees no unknown provider name; patchProviderIntoConfig injects credentials into generated config after binary writes it (models.providers.<name>)
…in --from-json payload The wizard sends both a top-level provider field and a providers array; clearing them to empty string still caused 'UNKNOWN PROVIDER' because nullclaw validates the name in both locations. Fix: replace custom provider entries with 'openai' (a known valid name) so the binary generates a valid base config, then patchProviderIntoConfig writes the real credentials into models.providers.<custom-name> afterwards. Also adds debug prints to confirm the detection path fires.
…m provider install - Add `model` field to CustomProvider; extract from wizard answers before stripping - patchProviderIntoConfig now sets agents.defaults.model.primary to <provider>/<model> - Remove the injected `openai` placeholder key from models.providers after binary runs
… install Custom providers backed by a local LLM may expose models whose vision probe never returns an error response — instead it hangs indefinitely, which blocks the nullclaw gateway HTTP handler during startup and causes the nullhub supervisor health-check to time out after 30 s. patchProviderIntoConfig now appends the installed model to agent.vision_disabled_models whenever a base_url is present, so the probe is skipped and the instance passes its first health check.
- ensureModelOptions now calls api.probeProviderModels(base_url, api_key) instead of api.getWizardModels for openai-compatible providers; the wizard list-models path calls nullclaw --list-models which does not support arbitrary base_url endpoints - modelKey now includes base_url so different custom endpoints with the same api_key get separate cache entries - server.zig: log the actual error when State.load() fails before falling back to empty state, preventing silent data loss
- WizardRenderer: openai-compatible providers (with base_url) are now validated via probeProviderModels HTTP probe instead of the nullclaw binary; nullclaw does not recognise the 'openai-compatible' provider name so it always returned live_ok=false, blocking the NEXT button - WizardRenderer: import OPENAI_COMPATIBLE_VALUE from providers.ts - Instance page: suppress provider health hint/warn while instance is not running (status != 'running') — stale 'instance_not_running' result from boot probe no longer shows as a provider error - Instance page: re-fetch provider health automatically when the instance transitions from any state to 'running'
nullclaw-dev-local panics with 'programmer bug caused syscall error: AGAIN' when a2a.enabled is false. The --from-json generator always emits enabled:false for fresh installs, so patchProviderIntoConfig now flips it to true while preserving all other a2a fields (url, name, version, etc.). Two tests cover the fix: one for the false→true mutation, one to confirm an already-true value is left unchanged. Also corrects the vision_disabled_models comment: the field is used at inference time to strip image markers, not to suppress the startup probe.
Contributor
Author
|
Follow-up from local repro/validation: the earlier managed-start diagnosis was wrong. The real managed-instance startup issue is tracked in #38. Root cause: managed instances were still launching a stale staged binary even after the local NullClaw build already contained the downstream daemon accept-loop fix. This PR remains the provider/model-probe work only. |
Contributor
Author
|
Follow-up from local repro/validation: the earlier managed-start diagnosis was wrong. The real managed-instance startup issue is tracked in #38. Root cause: managed This PR remains the provider/model-probe work only. |
OpenAI-compatible checks were timing out before healthy providers could respond through the probe path, which misreported working configs as failed.
This was referenced May 5, 2026
DonPrus
approved these changes
May 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /api/providers/probe-models?base_url=&api_key=so custom OpenAI-compatible endpoints can be validated and their/v1/modelslist fetched before savingnullclaw --list-models, which only understands built-in provider namesopenaiplaceholder, settingagents.defaults.model.primary, and populatingagent.vision_disabled_modelsfor inference-time image stripping10to30seconds in both instance and wizard flows so healthy custom endpoints are not misclassified as failedValidation
zig build test -Dbuild-ui=false --summary allNotes