Tool use dsrs

[AnthonyRonning]

No description provided.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch tool-use-dsrs

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Greptile Overview

Summary

This PR implements DSPy-style modules for tool use classification, migrating from manual prompt construction to a more structured approach using the DSRs library.

Key Changes

• New DSPy Integration: Added dspy-rs dependency from main branch and created custom OpenSecretAdapter that bridges DSPy's API with OpenSecret's existing completions infrastructure
• Intent Classification Module: Replaced manual prompt-based classification with IntentClassifier DSPy module that determines if user needs web search or chat
• Query Extraction Module: Added QueryExtractor DSPy module to extract clean search queries from natural language
• Maintained Infrastructure: Preserved existing billing, routing, retry logic, and auth handling by integrating DSPy calls through the centralized completions API
• Backward Compatibility: Marked legacy prompt templates as deprecated but kept them in place

Architecture

The implementation uses a custom adapter pattern where:

1. DSPy Signature defines input/output structure for classification and extraction
2. OpenSecretAdapter implements the DSPy Adapter trait but ignores the provided LM parameter
3. Instead, it creates OpenSecretLM that calls OpenSecret's existing get_chat_completion_response()
4. This ensures all LLM calls go through centralized billing, routing, and error handling

Notable Design Decisions

• Uses dummy LM instances to satisfy DSPy's API requirements, but the adapter ignores them
• Defaults to "chat" mode if classification fails (best-effort approach)
• Falls back to original user message if query extraction fails
• Uses fast model (llama-3.3-70b) with temperature=0 for deterministic classification

Confidence Score: 3/5

• Safe to merge with moderate risk - dependency on main branch and fallback behaviors need monitoring
• Score reflects the dependency stability risk (using main branch instead of pinned version), potential overhead from repeated dummy LM initialization, and silent fallback behaviors that could mask failures. The core logic is sound and integrates well with existing infrastructure, but production monitoring will be important to catch classification failures and unexpected DSPy API changes.
• Pay close attention to Cargo.toml (unstable dependency), dspy_modules.rs (repeated async initialization overhead), and handlers.rs (silent error fallbacks)

Important Files Changed

File Analysis

Filename	Score	Overview
Cargo.toml	4/5	Added dspy-rs dependency from main branch and anyhow error handling library
src/web/responses/dspy_adapter.rs	3/5	New custom DSPy adapter integrating with existing completions API infrastructure for billing and routing
src/web/responses/dspy_modules.rs	4/5	DSPy-style modules for intent classification and query extraction using OpenSecretAdapter
src/web/responses/handlers.rs	3/5	Replaced manual prompt construction with DSPy modules for intent classification and query extraction

Sequence Diagram

sequenceDiagram
    participant Client
    participant Handler as handlers.rs
    participant Classifier as IntentClassifier
    participant Extractor as QueryExtractor
    participant Adapter as OpenSecretAdapter
    participant LM as OpenSecretLM
    participant API as Completions API
    participant Tools as tools::execute_tool
    
    Client->>Handler: POST /v1/responses (user message)
    Handler->>Handler: validate_and_normalize_input()
    Handler->>Handler: build_context_and_check_billing()
    Handler->>Handler: persist_request_data()
    
    Handler->>Classifier: classify(message)
    Classifier->>Adapter: call(dummy_lm, signature, input)
    Adapter->>Adapter: format(signature, inputs)
    Adapter->>LM: call(messages)
    LM->>API: get_chat_completion_response()
    API-->>LM: FullResponse
    LM-->>Adapter: LMResponse
    Adapter->>Adapter: parse_response()
    Adapter-->>Classifier: Prediction{intent}
    Classifier-->>Handler: "web_search" or "chat"
    
    alt intent == "web_search"
        Handler->>Extractor: extract(message)
        Extractor->>Adapter: call(dummy_lm, signature, input)
        Adapter->>LM: call(messages)
        LM->>API: get_chat_completion_response()
        API-->>LM: FullResponse
        LM-->>Adapter: LMResponse
        Adapter-->>Extractor: Prediction{search_query}
        Extractor-->>Handler: search query string
        
        Handler->>Handler: send ToolCall to streams
        Handler->>Tools: execute_tool("web_search", args)
        Tools-->>Handler: tool output
        Handler->>Handler: send ToolOutput to streams
    end
    
    Handler->>API: get_chat_completion_response() (final response)
    API-->>Handler: CompletionChunk stream
    Handler-->>Client: SSE events with assistant response

Loading

_{6 files reviewed, 7 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: dependency points to main branch instead of a specific commit or version tag

Using main branch creates instability - any upstream changes will break your build without warning. Pin to specific commit hash or wait for v0.6.0 release.

Suggested change

	# Using main branch since v0.6.0 tag doesn't exist yet
	dspy-rs = { git = "https://github.com/krypticmouse/DSRs", branch = "main" }
	# DSPy for structured prompting and optimization
	dspy-rs = { git = "https://github.com/krypticmouse/DSRs", rev = "47ca636086403344b91fc4a90108f99a27101ae6" }

Prompt To Fix With AI

This is a comment left during a code review.
Path: Cargo.toml
Line: 81:82

Comment:
**logic:** dependency points to `main` branch instead of a specific commit or version tag

Using `main` branch creates instability - any upstream changes will break your build without warning. Pin to specific commit hash or wait for v0.6.0 release.

```suggestion
# DSPy for structured prompting and optimization
dspy-rs = { git = "https://github.com/krypticmouse/DSRs", rev = "47ca636086403344b91fc4a90108f99a27101ae6" }
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: dummy LM created with hardcoded "dummy_key" on every classifier instantiation

The adapter ignores this LM, but creating it with .await suggests it may perform initialization work (network calls, validation, etc.). Consider using lazy_static or caching to avoid repeated initialization overhead.

Also, if the LM builder validates the API key format, "dummy_key" might fail in future versions.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/dspy_modules.rs
Line: 48:54

Comment:
**style:** dummy LM created with hardcoded `"dummy_key"` on every classifier instantiation

The adapter ignores this LM, but creating it with `.await` suggests it may perform initialization work (network calls, validation, etc.). Consider using `lazy_static` or caching to avoid repeated initialization overhead.

Also, if the LM builder validates the API key format, `"dummy_key"` might fail in future versions.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: duplicate dummy LM initialization logic

Same concern as IntentClassifier - repeated .await calls may have overhead. Consider extracting to a shared get_dummy_lm() function or using a static instance.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/dspy_modules.rs
Line: 136:142

Comment:
**style:** duplicate dummy LM initialization logic

Same concern as IntentClassifier - repeated `.await` calls may have overhead. Consider extracting to a shared `get_dummy_lm()` function or using a static instance.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: token count conversions use as u32 without overflow checks

If token counts exceed u32::MAX (4.2B), this will silently wrap. While unlikely for current models, consider using .try_into() with proper error handling or document the assumption that token counts fit in u32.

DSPy v0.6.0 uses u32 for tokens, but OpenAI API returns i64, creating a type mismatch that could cause issues with future high-context models.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/dspy_adapter.rs
Line: 105:121

Comment:
**style:** token count conversions use `as u32` without overflow checks

If token counts exceed `u32::MAX` (4.2B), this will silently wrap. While unlikely for current models, consider using `.try_into()` with proper error handling or document the assumption that token counts fit in u32.

DSPy v0.6.0 uses u32 for tokens, but OpenAI API returns i64, creating a type mismatch that could cause issues with future high-context models.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: classification failure defaults to "chat" silently

If the DSPy call fails (network, quota, model error), the system defaults to chat mode without surfacing the error to the user. This means users won't get web search even when they need it. Consider logging at error! level (currently warn!) or adding metrics to track classification failures.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/handlers.rs
Line: 1208:1220

Comment:
**style:** classification failure defaults to `"chat"` silently

If the DSPy call fails (network, quota, model error), the system defaults to chat mode without surfacing the error to the user. This means users won't get web search even when they need it. Consider logging at `error!` level (currently `warn!`) or adding metrics to track classification failures.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: query extraction failure falls back to original user_text

If extraction fails, the entire user message (potentially multi-sentence) becomes the search query. This could result in poor search results. Consider either failing the search or extracting key terms with a simpler heuristic as backup.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/handlers.rs
Line: 1229:1239

Comment:
**style:** query extraction failure falls back to original `user_text`

If extraction fails, the entire user message (potentially multi-sentence) becomes the search query. This could result in poor search results. Consider either failing the search or extracting key terms with a simpler heuristic as backup.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: ignored lm parameter with comment explanation

This design creates tight coupling between DSPy's API expectations and OpenSecret's infrastructure. If DSPy changes its adapter API to require using the provided LM, this will break. Consider contributing upstream to make the LM parameter optional or adding integration tests to catch API changes early.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/web/responses/dspy_adapter.rs
Line: 196:197

Comment:
**style:** ignored `lm` parameter with comment explanation

This design creates tight coupling between DSPy's API expectations and OpenSecret's infrastructure. If DSPy changes its adapter API to require using the provided LM, this will break. Consider contributing upstream to make the LM parameter optional or adding integration tests to catch API changes early.

How can I resolve this? If you propose a fix, please make it concise.