Replace QueryPatternSearch with SearchManager across services

README.md

@@ -47,9 +47,32 @@ $response->results; // [['name' => 'Lowell Boyer', 'total_spent' => 3930.15], ..
 
 ## How It Works
 
-1. **Context Assembly** — Before the LLM sees the question, the agent retrieves relevant table metadata, business rules, similar query patterns, and past learnings. This assembled context is injected into the system prompt.
-2. **Agentic Tool Loop** — The LLM enters a tool-calling loop where it can introspect live schema, search for additional knowledge, execute SQL, and refine results iteratively.
-3. **Self-Learning** — When queries fail and the agent recovers, it saves what it learned. When queries succeed, it saves them as reusable patterns. Both feed back into step 1 for future queries.
+```mermaid
+flowchart TD
+    A[User Question] --> B[Retrieve Knowledge + Learnings]
+    B --> C[Reason about intent]
+    C --> D[Generate grounded SQL]
+    D --> E[Execute and interpret]
+    E --> F{Result}
+    F -->|Success| G[Return insight]
+    F -->|Error| H[Diagnose & Fix]
+    H --> I[Save Learning]
+    I --> D
+    G --> J[Optionally save as Knowledge]
+```
+
+The agent uses six context layers to ground its SQL generation:
+
+| # | Layer | What it contains | Source |
+|---|-------|-----------------|--------|
+| 1 | Table Usage | Schema, columns, relationships | `knowledge/tables/*.json` |
+| 2 | Human Annotations | Metrics, definitions, business rules | `knowledge/business/*.json` |
+| 3 | Query Patterns | SQL known to work | `knowledge/queries/*.json` and `*.sql` |
+| 4 | Learnings | Error patterns and discovered fixes | `save_learning` tool (on-demand) |
+| 5 | Runtime Context | Live schema inspection | `introspect_schema` tool (on-demand) |
+| 6 | Institutional Knowledge | Docs, wikis, external references | Custom tools (`agent.tools` config) |
+
+Layers 1–3 are loaded from the knowledge base into the system prompt. Layer 4 is built up over time as the agent learns from errors. Layers 5 and 6 are available on-demand — the LLM calls them during the tool loop when it needs live schema details or external context.
 
 ## Features
 

composer.json

@@ -29,7 +29,8 @@
     },
     "suggest": {
         "livewire/livewire": "Required for the chat UI",
-        "pgvector/pgvector": "Required for the pgvector search driver (vector similarity search)"
+        "pgvector/pgvector": "Required for the pgvector search driver (vector similarity search)",
+        "prism-php/relay": "Required for MCP server tool integration (relay config)"
     },
     "autoload": {
         "psr-4": {

config/sql-agent.php

@@ -170,6 +170,10 @@
         // Custom tool class names resolved from the container, e.g.:
         // [App\SqlAgent\MyCustomTool::class]
         'tools' => [],
+
+        // MCP server names (from config/relay.php) whose tools should be
+        // available to the agent. Requires prism-php/relay to be installed.
+        'relay' => [],
     ],
 
     /*
@@ -193,20 +197,15 @@
     |
     | Configure knowledge base settings.
     |
-    | Source options:
-    |   - 'database': Reads knowledge from the sql_agent_table_metadata,
-    |     sql_agent_business_rules, and sql_agent_query_patterns database tables.
-    |     Requires running `php artisan sql-agent:load-knowledge` first to import
-    |     JSON files into the database. Supports full-text search and is the
-    |     recommended option for production.
-    |   - 'files': Reads knowledge directly from JSON files on disk at the
-    |     configured path. No database import needed, but does not support
-    |     full-text search over knowledge.
+    | The path option sets the directory containing your JSON knowledge files.
+    | This path is used by the `sql-agent:load-knowledge` command to import
+    | files into the database. Knowledge is always read from the database at
+    | runtime — run `php artisan sql-agent:load-knowledge` after creating or
+    | changing knowledge files.
     |
     */
     'knowledge' => [
         'path' => env('SQL_AGENT_KNOWLEDGE_PATH', resource_path('sql-agent/knowledge')),
-        'source' => env('SQL_AGENT_KNOWLEDGE_SOURCE', 'database'),
     ],
 
     /*

docs/src/content/docs/getting-started/introduction.md

@@ -15,15 +15,18 @@ SqlAgent converts questions to SQL through multi-layer context assembly, an agen
 
 ### Context Assembly
 
-Before the LLM sees the question, the `ContextBuilder` retrieves and assembles five context layers into the system prompt:
-
-- **Semantic model** — Table metadata, column descriptions, and relationships from your [Knowledge Base](/sql-agent/guides/knowledge-base/)
-- **Business rules** — Metrics definitions, domain rules, and common gotchas
-- **Similar query patterns** — Previously validated queries that match the current question, retrieved via the active [search driver](/sql-agent/guides/drivers/)
-- **Relevant learnings** — Patterns the agent discovered from past errors and corrections
-- **Runtime schema** — Live database introspection for the tables most likely relevant to the question
-
-This happens automatically on every query — no manual prompt engineering required.
+Before the LLM sees the question, the `ContextBuilder` retrieves and assembles six context layers:
+
+| # | Layer | What it contains | Source |
+|---|-------|-----------------|--------|
+| 1 | Table Usage | Schema, columns, relationships | `knowledge/tables/*.json` |
+| 2 | Human Annotations | Metrics, definitions, business rules | `knowledge/business/*.json` |
+| 3 | Query Patterns | SQL known to work | `knowledge/queries/*.json` and `*.sql` |
+| 4 | Learnings | Error patterns and discovered fixes | `save_learning` tool (on-demand) |
+| 5 | Runtime Context | Live schema inspection | `introspect_schema` tool (on-demand) |
+| 6 | Institutional Knowledge | Docs, wikis, external references | [Custom tools](/sql-agent/guides/custom-tools/) (`agent.tools` config) |
+
+Layers 1–3 are loaded from the [Knowledge Base](/sql-agent/guides/knowledge-base/) and assembled into the system prompt automatically. Layer 4 is built up over time as the agent learns from errors. Layers 5 and 6 are available on-demand — the LLM calls them during the [tool loop](#agentic-tool-loop) when it needs live schema details or external context.
 
 ### Agentic Tool Loop
 

docs/src/content/docs/guides/configuration.md

@@ -195,11 +195,17 @@ You can add custom indexes by providing an `index_mapping` array in the driver c
 'database' => [
     // ...
     'index_mapping' => [
-        'custom_index' => \App\Models\CustomModel::class,
+        'faq' => \App\Models\Faq::class,
     ],
 ],
 ```
 
+Custom indexes are fully integrated into the search system:
+
+- The `search_knowledge` tool automatically exposes custom indexes to the LLM as additional type options.
+- The `ContextBuilder` searches custom indexes and includes matching results as "Additional Knowledge" in the system prompt.
+- Both `database` and `pgvector` drivers support custom indexes identically.
+
 Each model referenced in `index_mapping` must extend `Illuminate\Database\Eloquent\Model` and implement the `Knobik\SqlAgent\Contracts\Searchable` interface, which requires two methods:
 
 - `getSearchableColumns()` — Returns the column names to index for search.
@@ -238,6 +244,22 @@ You can extend the agent with your own tools by listing class names in the `tool
 
 Each class must extend `Prism\Prism\Tool` and is resolved from the Laravel container with full dependency injection support. See the [Custom Tools](/sql-agent/guides/custom-tools/) guide for detailed examples and best practices.
 
+### MCP Server Tools (Relay)
+
+If you have [Prism Relay](https://github.com/prism-php/relay) installed, you can bring tools from MCP servers into the agent by listing server names from `config/relay.php`:
+
+```php
+'agent' => [
+    // ... other options ...
+    'relay' => [
+        'weather-server',
+        'filesystem-server',
+    ],
+],
+```
+
+The `relay` key is silently ignored when `prism-php/relay` is not installed. See the [Custom Tools](/sql-agent/guides/custom-tools/#mcp-server-tools-relay) guide for full setup instructions.
+
 ## Learning
 
 SqlAgent can automatically learn from SQL errors and improve over time:
@@ -265,21 +287,17 @@ Schedule::command('sql-agent:prune-learnings')->daily();
 
 ## Knowledge
 
-Configure where SqlAgent reads knowledge from at runtime:
+Configure the knowledge base path:
 
 ```php
 'knowledge' => [
     'path' => env('SQL_AGENT_KNOWLEDGE_PATH', resource_path('sql-agent/knowledge')),
-    'source' => env('SQL_AGENT_KNOWLEDGE_SOURCE', 'database'),
 ],
 ```
 
-The `path` option sets the directory containing your JSON knowledge files. This path is used both when loading knowledge via `sql-agent:load-knowledge` and when the `files` source reads directly from disk.
-
-The `source` option controls how the agent loads knowledge at runtime:
+The `path` option sets the directory containing your JSON knowledge files. This path is used by the `sql-agent:load-knowledge` command to import files into the database.
 
-- **`database`** (default, recommended) — Reads from the `sql_agent_table_metadata`, `sql_agent_business_rules`, and `sql_agent_query_patterns` tables. You must run `php artisan sql-agent:load-knowledge` to import your JSON files first. Supports full-text search over knowledge.
-- **`files`** — Reads directly from JSON files on disk. No import step needed, but full-text search is not available.
+Knowledge is always read from the database at runtime — from the `sql_agent_table_metadata`, `sql_agent_business_rules`, and `sql_agent_query_patterns` tables. You must run `php artisan sql-agent:load-knowledge` after creating or changing knowledge files.
 
 ## Web Interface
 

docs/src/content/docs/guides/custom-tools.md

@@ -135,6 +135,63 @@ SqlAgent validates custom tools at boot time:
 
 These errors surface immediately when the application boots, not at query time, so misconfigurations are caught early.
 
+## MCP Server Tools (Relay)
+
+SqlAgent integrates with [Prism Relay](https://github.com/prism-php/relay) to bring tools from MCP (Model Context Protocol) servers into the agentic loop. This lets you connect external tool servers — filesystem access, API wrappers, code interpreters, or any MCP-compatible server — without writing custom PHP tool classes.
+
+### Installation
+
+Install the Relay package:
+
+```bash
+composer require prism-php/relay
+```
+
+Then publish and configure your MCP servers in `config/relay.php` following the [Relay documentation](https://github.com/prism-php/relay).
+
+### Registering MCP Server Tools
+
+List the MCP server names (as defined in `config/relay.php`) in the `agent.relay` array in `config/sql-agent.php`:
+
+```php
+'agent' => [
+    // ... other options ...
+    'tools' => [],
+
+    'relay' => [
+        'weather-server',
+        'filesystem-server',
+    ],
+],
+```
+
+At boot time, SqlAgent calls `Relay::tools($server)` for each configured server and registers all discovered tools alongside the built-in and custom tools. The LLM sees and can call all of them.
+
+:::tip
+Relay tools are dynamically discovered `Tool` instances — you don't need to create PHP classes for them. Just configure the MCP server in `config/relay.php` and reference it by name in the `relay` array.
+:::
+
+:::note
+If `prism-php/relay` is not installed, the `relay` config key is silently ignored. This means you can ship a config that references Relay servers without requiring the package — useful for shared config across environments where only some have Relay installed.
+:::
+
+### Combining Custom Tools and Relay
+
+Custom tools and Relay tools can be used together. They all end up in the same tool registry and are passed to the LLM equally:
+
+```php
+'agent' => [
+    'tools' => [
+        \App\SqlAgent\CurrentDateTimeTool::class,
+    ],
+    'relay' => [
+        'weather-server',
+    ],
+],
+```
+
+If a Relay tool has the same name as a built-in or custom tool, it will overwrite the previous registration (last write wins).
+
 ## Tips
 
 - **Keep tools focused.** Each tool should do one thing well. Prefer two small tools over one tool with a `mode` parameter.

docs/src/content/docs/guides/knowledge-base.md

@@ -214,5 +214,5 @@ php artisan sql-agent:load-knowledge --path=/custom/knowledge/path
 ```
 
 :::caution
-When using the default `database` knowledge source, you **must** run this command after creating or changing knowledge files. The agent reads from the database at runtime, not directly from disk.
+You **must** run this command after creating or changing knowledge files. The agent always reads knowledge from the database at runtime, not directly from disk.
 :::

docs/src/content/docs/guides/multi-database.md

@@ -100,7 +100,7 @@ Each connection can define its own table and column restrictions:
 Restrictions are enforced at every layer:
 
 - Schema introspection (listing tables, inspecting columns)
-- Semantic model loading (table metadata from files or database)
+- Semantic model loading (table metadata from database)
 - SQL execution (queries referencing denied tables are rejected)
 
 ## Web Interface
@@ -125,7 +125,7 @@ Each step is visible in the streaming UI as a separate tool call with its connec
 
 ## Knowledge Loading
 
-When using the `database` knowledge source (the default), table metadata is scoped per connection using the `connection` field in each JSON knowledge file. When you run `sql-agent:load-knowledge`, the loader reads the `connection` field from each table JSON file and stores it in the database alongside the metadata.
+Table metadata is scoped per connection using the `connection` field in each JSON knowledge file. When you run `sql-agent:load-knowledge`, the loader reads the `connection` field from each table JSON file and stores it in the database alongside the metadata.
 
 ### Tagging Knowledge Files
 
@@ -156,14 +156,6 @@ Files without a `connection` field default to `"default"` and are included for a
 Use the same logical names in your JSON files that you use as keys in the `database.connections` config. For example, if your config has `'crm' => [...]`, set `"connection": "crm"` in the corresponding knowledge files.
 :::
 
-### File-Based Knowledge Source
-
-When using the `files` knowledge source, you can also add a `connection` field to your JSON files. Tables with a matching connection (or no connection field) are included when building context for each database. Tables tagged with a different connection name are filtered out.
-
-:::caution
-For the best multi-database experience, use the `database` knowledge source. It provides full-text search support and precise per-connection filtering via the `connection` column.
-:::
-
 ## Limitations
 
 - **No cross-database JOINs.** The agent runs separate queries and combines results programmatically.

docs/src/content/docs/guides/recommended-configs.md

@@ -34,9 +34,6 @@ SQL_AGENT_MAX_ITERATIONS=15
 # Learning — enabled with auto error capture
 SQL_AGENT_LEARNING_ENABLED=true
 SQL_AGENT_AUTO_SAVE_ERRORS=true
-
-# Knowledge — database source for full-text search
-SQL_AGENT_KNOWLEDGE_SOURCE=database
 ```
 
 ### pgvector Database Connection
@@ -92,9 +89,6 @@ SQL_AGENT_MAX_ITERATIONS=10
 # Learning — enabled
 SQL_AGENT_LEARNING_ENABLED=true
 SQL_AGENT_AUTO_SAVE_ERRORS=true
-
-# Knowledge — database source
-SQL_AGENT_KNOWLEDGE_SOURCE=database
 ```
 
 No embeddings configuration is needed since the database search driver uses native full-text search instead of vector embeddings.

docs/src/content/docs/reference/commands.md

@@ -39,7 +39,7 @@ After running this command, generate embeddings for your existing knowledge base
 
 ## `sql-agent:load-knowledge`
 
-Import knowledge files from disk into the database. Required when using the default `database` knowledge source.
+Import knowledge files from disk into the database. Required after creating or changing knowledge files.
 
 ```bash
 php artisan sql-agent:load-knowledge

src/Data/Context.php

@@ -16,14 +16,17 @@ public function __construct(
         public Collection $queryPatterns,
         /** @var Collection<int, mixed> */
         public Collection $learnings,
-        public ?string $runtimeSchema = null,
-    ) {}
+        /** @var Collection<int, array<string, mixed>> */
+        public ?Collection $customKnowledge = null,
+    ) {
+        $this->customKnowledge ??= collect();
+    }
 
     public function toPromptString(): string
     {
         $sections = [];
 
-        // Layer 1: Semantic Model (Table Usage)
+        // Layer 1: Semantic Model
         if ($this->semanticModel) {
             $sections[] = $this->formatSection('DATABASE SCHEMA', $this->semanticModel);
         }
@@ -41,17 +44,29 @@ public function toPromptString(): string
             $sections[] = $this->formatSection('SIMILAR QUERY EXAMPLES', $patterns);
         }
 
-        // Layer 5: Learnings
+        // Layer 4: Learnings
         if ($this->learnings->isNotEmpty()) {
             $learnings = $this->learnings
                 ->map(fn ($l) => "- {$l['title']}: {$l['description']}")
                 ->implode("\n");
             $sections[] = $this->formatSection('RELEVANT LEARNINGS', $learnings);
         }
 
-        // Layer 6: Runtime Schema
-        if ($this->runtimeSchema) {
-            $sections[] = $this->formatSection('RUNTIME SCHEMA INSPECTION', $this->runtimeSchema);
+        // Layer 5: Custom Knowledge
+        if ($this->customKnowledge->isNotEmpty()) {
+            $knowledge = $this->customKnowledge
+                ->map(function (array $item) {
+                    $parts = [];
+                    foreach ($item as $key => $value) {
+                        if ($value !== null && $value !== '') {
+                            $parts[] = "{$key}: {$value}";
+                        }
+                    }
+
+                    return '- '.implode(' | ', $parts);
+                })
+                ->implode("\n");
+            $sections[] = $this->formatSection('ADDITIONAL KNOWLEDGE', $knowledge);
         }
 
         return implode("\n\n", $sections);
@@ -72,11 +87,6 @@ public function hasLearnings(): bool
         return $this->learnings->isNotEmpty();
     }
 
-    public function hasRuntimeSchema(): bool
-    {
-        return ! empty($this->runtimeSchema);
-    }
-
     public function getQueryPatternCount(): int
     {
         return $this->queryPatterns->count();
@@ -93,6 +103,6 @@ public function isEmpty(): bool
             && empty($this->businessRules)
             && $this->queryPatterns->isEmpty()
             && $this->learnings->isEmpty()
-            && empty($this->runtimeSchema);
+            && $this->customKnowledge->isEmpty();
     }
 }