Smarter symbol-level reindexing, ReFRAG memory server alignment, git commit lineage, dev-remote perms + index speed improvements + worktree deup

[voarsh2]

Summary

This PR fixes a ReFRAG collection configuration mismatch between the memory server and indexer, introduces an opt‑in, symbol‑aware smart reindexing path that reuses embeddings for unchanged code while preserving line numbers, and adds optional git commit lineage indexing wired through the VS Code extension and remote upload pipeline. It also hardens workspace state and symbol cache handling for shared volumes in dev‑remote.

---

Problem

• ReFRAG collection mismatch

  • Memory server created collections with only [dense, lex] vectors, ignoring REFRAG_MODE.
  • Indexer expected [dense, lex, mini] vectors when ReFRAG was enabled, causing Not existing vector name: mini errors.
  • Qdrant does not support adding new vector names to existing collections via update_collection().
  • update_collection() calls were effectively no‑ops, leading to indexing failures and retry loops.
  • Collection recreation attempts were failing / not preserving data, causing the indexer to get stuck.

• Inefficient full‑file reindexing

  • Any change to a file triggered a full delete+reinsert of all chunks and re‑embedding, even when most symbols were unchanged.
  • Pseudo/tag generation ran for entire files repeatedly, increasing GLM usage.
  • Line number accuracy is critical, but we had no symbol‑level strategy to reuse embeddings safely.

• No integrated git commit lineage pipeline

  • There was no end‑to‑end path from a workspace’s git history into Qdrant as first‑class commit points.
  • Commit lineage metadata (lineage_goal, lineage_symbols, lineage_tags) was only available in ad‑hoc CLI experiments, not through the dev‑remote watcher + upload service.
  • The VS Code extension had no way to trigger or scope git history uploads (max commits, --since, force).
  • We had no durable, queryable commit index that could power search_commits_for / change_history_for_path in normal workflows.

• Git history pipeline and initial inefficiency

  • There was no end‑to‑end git history pipeline wired into the dev‑remote flow:
    • Upload clients didn’t emit git history manifests in bundles.
    • upload_service / watch_index weren’t extracting or ingesting git history into Qdrant.
  • When we first wired git history into the upload → upload_service → watcher → ingest_history.py pipeline, the initial client implementation:
    • Recomputed a full REMOTE_UPLOAD_GIT_MAX_COMMITS window from HEAD on each run.
    • Re‑embedded and re‑summarized every commit in that window whenever HEAD changed, with no notion of a “last ingested head”.
    • Had no extension‑level git cache; small new commits could still trigger hundreds of redundant decoder + embedding calls.
  • The remote side already stored git bundles under .remote-git, but this store was only used as a drop location for manifests, not to guide incremental ingestion or provide guarantees about what had been processed.

• Dev‑remote permissions and cache robustness

  • Mixed UIDs (root vs non‑root) and bind‑mounted volumes caused .codebase permission errors, especially for symbol caches.
  • Corrupt or empty cache.json and symbol cache files could break indexing, with poor recovery behavior.

---

Solution

1. ReFRAG collection configuration + memory backup/restore

• Respect REFRAG_MODE in memory server

  • Add REFRAG_MODE handling to the memory server’s _ensure_collection() so new collections are created with the same vector layout the indexer expects:
    • ReFRAG off: [dense, lex].
    • ReFRAG on: [dense, lex, mini].

• Safe collection recreation with memory preservation

  • Implement a backup/restore path in ingest_code.py’s ensure_collection():
    • Before any destructive recreation, export “memories” defined as points without file_path in their payload.
    • Recreate the collection with the desired vector configuration (including mini when ReFRAG is enabled).
    • Restore memories into the new collection, allowing partial vector support (e.g., older memories without mini).
  • Add proper error handling and logging around collection recreation so failures are visible and don’t create silent loops.

Result: Memory server and indexer now agree on vector names from the start; if we ever do need to recreate a collection, user memories are preserved and upgraded safely.

---

2. Symbol‑aware smart reindexing (opt‑in via SMART_SYMBOL_REINDEXING)

• Symbol cache and change detection

  • Add symbol‑level cache APIs in workspace_state.py:
    • _get_symbol_cache_path(file_path) → per‑repo .codebase/repos/<repo>/symbols_<hash>.json using the same repo detection as other state helpers.
    • get_cached_symbols(file_path) / set_cached_symbols(file_path, symbols, file_hash) → store symbol metadata + content hashes.
    • get_cached_pseudo(...), set_cached_pseudo(...), update_symbols_with_pseudo(...) → cache pseudo/tags per symbol without creating empty symbol files.
    • compare_symbol_changes(old_symbols, new_symbols) → compute (unchanged_symbols, changed_symbols) using content hashes.
    • remove_cached_symbols(file_path) → cleanly remove symbol cache when a file is deleted (used by watcher).

• Smart reindex pipeline (new code path)

  • Implement process_file_with_smart_reindexing(...) in ingest_code.py:
    • Extract current symbols and load cached symbols.
    • Compute unchanged vs changed symbol IDs.
    • Load existing Qdrant points for the file and index them by (symbol_id, code_text) for reuse.
    • Re‑chunk the file using the existing chunking strategy (micro/semantic/line, as configured).
    • For each chunk:
      • Use should_process_pseudo_for_chunk(...) + generate_pseudo_tags(...) + set_cached_pseudo(...) to only run GLM for changed symbols/chunks.
      • Attempt to reuse an existing embedding when (symbol_id, code_text) matches; otherwise re‑embed.
    • Replace all points for the file via delete_points_by_path(...) + upsert_points(...), preserving line numbers.
    • Update symbol + file‑hash caches only on success.
    • Emit detailed logs:
      • unchanged_symbols, changed_symbols, and per‑file summary:
        chunks=X, reused_points=Y, embedded_points=Z.

• Integration and gating

  • Indexer (index_repo)

    • Under SMART_SYMBOL_REINDEXING and should_use_smart_reindexing(...), call process_file_with_smart_reindexing(...).
    • On "success" → count as indexed and skip legacy path.
    • On "failed" → fall back to index_single_file(...).
    • Continue to use skip_unchanged and symbol change heuristics to limit smart reindexing only when it’s beneficial.

  • Watcher (watch_index.py)

    • On file deletion:
      • Remove file‑hash cache and symbol cache (remove_cached_symbols) to keep caches consistent.
    • On file changes in _process_paths(...):
      • If _smart_symbol_reindexing_enabled():
        • Read file text, detect language, compute hash.
        • Call should_use_smart_reindexing(path, file_hash) and interpret smart_reason.
        • Support bootstrap: if smart_reason == "no_cached_symbols", run smart reindex once to seed the cache.
        • Log decisions: [SMART_REINDEX][watcher] Using {smart/full} ... ({reason}).
        • On success → done; on failure → fall back to index_single_file(...).

  • Feature flag

    • Add SMART_SYMBOL_REINDEXING to .env.example (default 0):

      # Smarter re-indexing for symbol cache, reuse embeddings and reduce decoder/pseudo tags to re-index
      SMART_SYMBOL_REINDEXING=0

Result: When enabled, both batch indexer and watcher reuse embeddings for unchanged code at a symbol level, drastically reducing re‑embedding and pseudo costs, while keeping a clean escape hatch back to the legacy path.

---

3. Git commit history ingestion and lineage search (opt‑in)

• Upload clients: manifest + cache

  • Extend remote_upload_client.py and standalone_upload_client.py with _collect_git_history_for_workspace(...):
    • Read REMOTE_UPLOAD_GIT_MAX_COMMITS, REMOTE_UPLOAD_GIT_SINCE, REMOTE_UPLOAD_GIT_FORCE from the environment.
    • When max_commits > 0, collect recent commits and write metadata/git_history.json into each bundle.
  • Maintain a git history cache at:
    • <workspace>/.context-engine/git_history_cache.json
    • Fields: last_head, max_commits, since, updated_at.
  • Incremental history:
    • First run (or REMOTE_UPLOAD_GIT_FORCE=1): git rev-list --no-merges [--since] HEAD truncated to max_commits.
    • Subsequent runs (no force, cache present):
      • Use git rev-list --no-merges [--since] <last_head>..<HEAD> so only new commits since the last ingested head are collected.
      • Avoids re‑embedding/re‑summarizing a large window (e.g. 500 commits) for every small change to HEAD.

• Upload service: .remote-git manifest store

  • upload_service.py now:
    • Extracts metadata/git_history.json from incoming bundles.
    • Writes them under:
      • <workspace>/.remote-git/git_history_<bundle_id>.json
    • This forms a durable, append‑only store of git bundles for that workspace.

• Watcher: manifest routing

  • watch_index.py:
    • Explicitly enqueues .remote-git/*.json files, bypassing normal CODE_EXTS filters.
    • In _process_paths(...), routes these to _process_git_history_manifest(...).
  • _process_git_history_manifest(...):
    • Spawns ingest_history.py --manifest-json /work/.../.remote-git/git_history_<bundle_id>.json as a subprocess.
    • Propagates COLLECTION_NAME, QDRANT_URL, and REPO_NAME into the child process environment.
    • Logs a [git_history_manifest] line for observability.

• Ingest script: manifest mode + lineage summaries

  • ingest_history.py:
    • Supports --manifest-json mode that:
      • Reads a git history manifest (git_history.json).
      • For each commit, builds a text payload from message, files, and diff.
      • Embeds it via fastembed.
      • Uses stable_id(commit_id) as the Qdrant point id for idempotent upserts.
    • Uses generate_commit_summary(...) to produce:
      • lineage_goal, lineage_symbols, lineage_tags.
      • Gated by REFRAG_COMMIT_DESCRIBE=1 and decoder env (REFRAG_DECODER, REFRAG_RUNTIME, GLM/llamacpp configs).
    • Ensures scripts is importable when run as a subprocess by adding the project root to sys.path.

• MCP tools and queries

  • Commit points are stored in the same collection as code, with:
    • kind="git_message" (or equivalent metadata), commit_id, author_name, authored_date, files, message, lineage_*.
  • Existing MCP tools:
    • search_commits_for and change_history_for_path can now:
      • Filter and rank commits by message and lineage tags/goals.
      • Power “when/why did behavior X change?” workflows described in the commit indexing docs.

• VS Code extension wiring

  • The context-engine-uploader extension:
    • Adds settings:
      • contextEngineUploader.gitMaxCommits (default 500).
      • contextEngineUploader.gitSince (empty by default).
      • A command to force git history upload (sets REMOTE_UPLOAD_GIT_FORCE=1 for a run).
    • Wires these settings into the env for remote_upload_client.py / standalone_upload_client.py.
    • Documents the git history behavior and settings in the extension README.
  • .env.example:
    • Documents REMOTE_UPLOAD_GIT_MAX_COMMITS, REMOTE_UPLOAD_GIT_SINCE, and REFRAG_COMMIT_DESCRIBE=1 as the toggle for commit lineage summarization.

Result: We now have an end‑to‑end, opt‑in pipeline for git commit history and lineage indexing, driven by the VS Code extension and upload clients, with incremental behavior and MCP‑friendly commit points in Qdrant.

---

4. Workspace state + dev‑remote permissions hardening

• State and symbol cache permissions

  • _atomic_write_state(...) now chmods state.json / cache.json to 0664 so multiple processes (upload, watcher, indexer) can update them on shared volumes.
  • set_cached_symbols(...) ensures symbol cache files are created with 0664 and parent dirs with 0775, improving multi‑process access on .codebase volumes.

• Dev‑remote compose alignment

  • Run indexer-dev-remote as user: "1000:1000", matching watcher and MCP services, so they can all write to .codebase without root‑owned artifacts.
  • Keep upload-service-dev-remote as user: "0:0" only where needed for Windows /work bind mounts; it no longer has to own .codebase.
  • Combined with the chmod changes above, this resolves the recurring symbol cache permission errors in dev‑remote.

---

Impact

• ReFRAG / memory server

  • Collections created by the memory server now match the indexer’s expectations:
    • Correct vector set (dense, lex, mini where applicable).
  • Eliminates:
    • Not existing vector name: mini and HTTP 400 errors.
    • Silent update_collection() failures that previously left collections in a half‑configured state.
  • Enables ReFRAG mode to work correctly without manual collection recreation.
  • Memory backup/restore preserves user “memories” (points without file_path) across any future collection config changes.

• Smart symbol reindexing

  • Opt‑in via SMART_SYMBOL_REINDEXING=1; default is off for safety.
  • When enabled:
    • Reuses existing embeddings for unchanged chunks mapped to unchanged symbols (no re-embedding for those chunks).
    • Greatly reduces GLM decoder/pseudo calls for small edits by only regenerating pseudo for changed symbols/chunks.
    • Still rebuilds the full set of points for each changed file (delete + insert), so line numbers and chunk boundaries are always based on the latest text.
    • Provides rich [SMART_REINDEX] logging for observability and debugging.

• Git commit lineage

  • Optional git history indexing, controlled by:
    • Extension settings (gitMaxCommits, gitSince, “Upload Git History” / force command).
    • Env flags (REMOTE_UPLOAD_GIT_MAX_COMMITS, REMOTE_UPLOAD_GIT_SINCE, REMOTE_UPLOAD_GIT_FORCE).
    • Lineage summarization toggle via REFRAG_COMMIT_DESCRIBE + decoder env.
  • Incremental uploads via last_head..HEAD drastically reduce redundant commit reprocessing while remaining idempotent in Qdrant (stable ids per commit_id).
  • Commit points are now first‑class citizens in the collection, enabling better lineage workflows via MCP tools.

• Operational robustness

  • Fewer dev‑remote permission flakes on .codebase and symbol caches.
  • workspace_state.py behaves better in multi‑process and multi‑repo setups.
  • Clearer logs and error paths around collection creation/recreation, smart reindex decisions, and git history ingest.

---

Backwards Compatibility / Rollout

• Feature flags

  • SMART_SYMBOL_REINDEXING is default 0; legacy indexing path remains unchanged when the flag is off.
  • ReFRAG behavior is driven by REFRAG_MODE; when disabled, we still use only [dense, lex].
  • Git commit lineage is opt‑in:
    • Disabled unless REMOTE_UPLOAD_GIT_MAX_COMMITS (or the extension setting) is non‑zero.
    • Lineage summaries only generated if REFRAG_COMMIT_DESCRIBE=1 and decoder env is configured.

• Existing collections

  • Existing collections continue to work; new ReFRAG‑aware logic only kicks in when REFRAG_MODE is enabled and/or when a collection must be recreated.
  • Memory backup/restore ensures we do not lose user memories during controlled recreations.

• Rollout plan

  • Start with dev‑remote / non‑critical environments:
    • Enable SMART_SYMBOL_REINDEXING=1.
    • Configure git history settings in the extension and .env as needed.
    • Verify logs (reused_points vs embedded_points, symbol cache behavior, git ingest logs) and Qdrant payloads (line numbers, symbol metadata, lineage_*).
  • Once validated, consider enabling smart reindexing and git history indexing in a canary cluster before broader rollout.

---

Testing

• Dev-remote (code indexing)

  • Ran full repo indexing with ReFRAG enabled; verified:
    • No mini vector missing errors.
    • Memory server and indexer agree on vector configuration.
  • Confirmed SMART_SYMBOL_REINDEXING=0 uses legacy behavior; =1 triggers smart reindex logs and reduces embedding counts on small edits.
  • Verified watcher path:
    • Smart reindex used when symbol cache exists or bootstrap is needed.
    • Fallback to full index_single_file(...) on any smart reindex failure.
    • Symbol caches and file-hash caches removed on file deletes.

• Dev-remote (git lineage)

  • From the VS Code extension:
    • Configured gitMaxCommits and gitSince, and used the “Upload Git History” / force command.
    • Observed upload clients writing metadata/git_history.json and maintaining .context-engine/git_history_cache.json.
    • Confirmed upload_service writes .remote-git/git_history_<bundle_id>.json.
    • Confirmed watch_index.py logs [git_history_manifest] and launches ingest_history.py with the correct collection and repo name.
  • Ingest:
    • Verified ingest_history.py --manifest-json imports correctly in the container environment and upserts commit points using stable_id(commit_id).
    • Observed non‑empty lineage_goal / lineage_tags when REFRAG_COMMIT_DESCRIBE=1 and decoder env is configured.

• Qdrant / MCP inspection

  • Scrolled points for selected files before/after small edits:
    • Same or reduced point counts.
    • Updated chunks have correct line ranges and metadata.
    • Unchanged chunks reuse prior IDs/vectors where expected.
  • Used MCP tools (search_commits_for, change_history_for_path) to:
    • Retrieve commits with populated lineage_goal, lineage_symbols, lineage_tags.
    • Confirm that extension‑driven git uploads produce queryable commit history wired into the same collection as code.

---

[voarsh2]

Investigating windows related host_path bug appending container "/work"+windows path.... and seemingly empty repo_search results returning no code snippets...
Changed back to in draft until I can identify and fix