Proposal: Remote deployment + multi-repo collections (watcher/indexer + MCP) compatibility

[voarsh2]

Summary

• Add first-class support to run the stack remotely (e.g., Kubernetes) with minimal code changes
• Enable multi-repo operation via “collections,” with shared root metadata and per-repo metadata
• Ensure watcher/indexer and MCP tools work across collections and can be queried by collection
• Provide Kubernetes YAML for deployment and an optional Python uploader utility

Motivation

• I use Kubernetes clusters with stronger hardware than my thin client for remote dev
• Running MCP services remotely lets me develop from anywhere while delegating heavy work to the cluster
• Multi-repo support is needed for large monorepos and multiple project repos in the same environment

Scope (and non-goals)

• Scope: watcher/indexer, MCP tools, metadata schema, hybrid search param, remote deploy manifests, uploader utility
• Non-goals: changing core retrieval algorithms or replacing existing local dev workflow

Proposed changes

Collections

• Introduce a collection identifier to group repo-specific indexes and memory
• Each repo registers under a collection; MCP calls accept collection as a required or optional param

Metadata layout

• Root-level metadata
• Per-repo metadata

Watcher/indexer

• Support multiple collections/repositories concurrently
• Indexer watches paths mounted under collections//
• Minimal changes: add collection awareness to existing path registry and event handlers

MCP tools

• accept collection and project_id to ensure persistence independent of workspace paths
• accept collection and repo path/glob; tag chunks with collection + repo identity
• accept collection param (hybrid_search accepts collection) to scope retrieval
• Return citations as [collection/repo:file_path:start_line-end_line]

Hybrid search

• Add collection to query params
• Default: collection required; optional “all” collection for cross-collection queries if enabled by config

Python uploader utility

• Small CLI that uploads files into an “upload” pod/volume
• Places files under collections// so watcher sees changes across repos

Compatibility

• Preserve original tool interfaces; add collection param with sane defaults
• If collection is omitted, fallback to a default collection and maintain current behavior
• No breaking changes to existing single-repo/local workflows

Something you'd accept in a PR after review?

[John-Donalson]

yes absolutely! Any contributions are welcome

[m1rl0k]

I added you as contributor to repo as well

[voarsh2]

Great. I am prototyping locally - testing and will then open a PR when ready. Talk soon. :)

[m1rl0k]

@voarsh2 updated, go check it out

[voarsh2]

@voarsh2 updated, go check it out

Oh no! I'm like 90% done lol- doing edge case testing, etc for the multi repo support.

I'll stash and test - but, my fork was out of date on the progress I was making.

I'll check out what you've done.... see if I can use any of my uncommitted work....

[m1rl0k]

feel free to overwrite, was just trying to get the basics up

[voarsh2]

I finished all my work - all tested - and spend longer than I would like on merging upstream. :D
Finishing this up now as we speak, updating some tests. :)

[voarsh2]

@m1rl0k please review #20 when you can - would be nice if it's merged. :)

[voarsh2]

I was thinking of additionally extending the remote upload code and upload service to accept a json blob of the git commit history - wasn't sure how useful the current implementation of it is - have you found the commits history being indexed useful?

[m1rl0k]

I have not tested it much, its still in early stages. first step would to be git diffs in with relations to commit history. the concern is with giant prs and commits the amount of load the model might do, and would that be parallel to the semantic searching the current context.

So, feel free to experiment; this is something I would like to figure out!