Pseudo/tag backfill - indexer base index fast pass, faster index path for unchanged files

[voarsh2]

Bugs fixes :

Fixes some bugs in watch_index, namely:

• Avoid deriving a root-level "/work-" collection in multi-repo mode
• Resolve per-file Qdrant collections via get_collection_for_file for all data ops
• Fix on_deleted and move/rename delete paths to use repo-specific collections instead of the watcher’s default_collection

Improvements:

Pseudo / tags passthrough in hybrid_search

• Ensure index‑time pseudo and tags metadata are carried through in hybrid_search results:

  • In run_hybrid_search, we now extract pseudo / tags from the Qdrant payload/metadata and attach them to each returned item.
  • This is a schema enrichment only; the core dense + lexical + heuristic scoring pipeline is unchanged.

• Why this matters:

  • Downstream consumers (e.g. repo_search rerankers, IDE integrations, MCP clients) can now:
    • Display GLM/LLM‑generated summary labels and tags alongside each candidate,
    • Optionally incorporate pseudo / tags into their own secondary scoring or UI logic.
  • This makes it possible to build higher‑level rerankers and UIs that understand index‑time labels without having to re‑query Qdrant for metadata.

• Scope:

  • No behavior change to base ranking in hybrid_search itself; we intentionally keep pseudo / tags as metadata inputs for higher layers, not as mandatory ranking signals.

repo_search mode knob (code_first / balanced / docs_first)

• Add an explicit mode knob to the repo_search MCP tool:

  • Supported values: code_first, balanced (default), docs_first.
  • Plumbed through repo_search → run_hybrid_search (in scripts/hybrid_search.py) and exposed via repo_search_compat.

• Make run_hybrid_search implementation/doc weighting mode‑aware:

  • code_first (default / no mode):
    • Full IMPLEMENTATION_BOOST.
    • Full DOCUMENTATION_PENALTY.
    • Slight TEST_FILE_PENALTY to prefer implementation over tests.
  • balanced:
    • Keeps implementation boost.
    • Halves the structural doc penalty to let strong docs compete more fairly when they’re clearly relevant.
  • docs_first:
    • Reduces implementation boost.
    • Disables structural documentation penalty so docs can surface before code when they’re equally relevant.
  • Documentation penalties are strictly structural (README/docs/.md/etc), never query‑phrase based.

• MCP‑side, keep result shaping mode‑aware in repo_search:

  • code_first groups core implementation code, other code, then docs when ranking.
  • docs_first inverts that: docs first, then implementation/test code.
  • A simple post‑processing shim ensures at least N “core code” hits appear in the top‑K for code_first, tunable via:
    • REPO_SEARCH_CODE_FIRST_MIN_CORE
    • REPO_SEARCH_CODE_FIRST_TOP_K

• Empirical behavior (manual stress tests):

  • Queries with mixed code + docs (e.g. “how does hybrid search work”, “remote upload delta client and watcher”, “context answer tool usage and architecture”):
    • code_first reliably pulls .py implementation files to the top and pushes .md docs down, while still keeping high‑value docs in the tail of the top‑K.
    • docs_first flips the ordering: doc pages (README, MCP_API, CLAUDE.example, GETTING_STARTED, etc.) dominate the top‑K, with implementation files and tests following.
    • The default/balanced mode sits between these two, returning a blended set (a mix of implementation + docs), suitable when the caller doesn’t have a strong preference.
  • Purely docs‑oriented queries (e.g. “Kubernetes deployment configuration and documentation”) converge across all three modes, as expected, because there are effectively no strong code competitors.
  • Overall, mode behaves as intended:
    • When there is real competition between code and docs, it meaningfully tilts the top‑K toward the selected side.
    • When the corpus is effectively “all code” or “all docs” for a query, it doesn’t introduce noise.

• Usage guidance:

  • Use mode="code_first" for agents or workflows that need “where is this implemented?” answers, but still want docs as a fallback in the same call.
  • Use mode="docs_first" when you primarily want conceptual/usage explanations from documentation and only occasionally need to dive into code.
  • For “only code” or “only docs” scenarios, prefer hard filters (path_glob / not_glob) and use mode as a softer preference layer on top.

Perf:

Implements a mechanism to skip re-indexing files based on file size and modification time.

This optimization, enabled by the INDEX_FS_FASTPATH environment variable, significantly speeds up indexing, particularly when dealing with large repositories or frequent re-indexing operations where file contents may not have changed. The logic retrieves file metadata (size and mtime) from the cache and compares it with the current file's metadata. If they match, the file is skipped, avoiding unnecessary re-reading and processing.

The change also updates the cache to store file size and mtime along with the file hash.
When fast-fs is enabled, this commit refreshes the file hash cache with size/mtime information during file skipping. This ensures the cache remains up-to-date even when files are skipped due to unchanged content, and enhances the fast-fs
performance.

• Add an INDEX_FS_FASTPATH-gated precheck in index_repo that walks files
  using cache.json fs metadata (size/mtime) and, when every file matches,
  exits early before model construction and Qdrant client setup, making
  true no-change runs much cheaper.
• Leave behavior unchanged for any new/changed/uncached files or entries
  missing size/mtime metadata (these still fall back to the original
  full index path).
• Add a TODO above the collection health check to note that these
  expensive Qdrant probes should eventually be split into a dedicated
  "health-check-only" mode, so "nothing changed" runs can remain fast
  while still offering an explicit way to validate collections.

[voarsh2]

I am debating the usefulness of "mode" mcp arg for code_first/docs first... might rip it out after experimenting some more.