Conversation
Introduces a tree cache to ASTAnalyzer to avoid re-parsing unchanged files, improving performance for repeated analysis. Enhances vector projection in scripts/ingest/vectors.py by using numpy when available for significant speedup, with a fallback to pure Python. Updates Qdrant upsert logic to support async/sync modes via parameter or environment variable, and adds a flush_upserts utility to ensure data persistence after async upserts.
🤖 Augment PR SummarySummary: Performance-focused update to reuse parsed ASTs, speed up mini-vector projection, and optionally run Qdrant upserts asynchronously. Changes:
Technical Notes: Default behavior remains synchronous unless async mode is enabled; projection matrices are cached and NumPy uses vectorized matmul + L2 normalization. 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
scripts/ast_analyzer.py
Outdated
| path = Path(file_path) if file_path else None | ||
|
|
||
| # Try to get cached tree (only for real files, not in-memory content) | ||
| if self._tree_cache and path and path.exists(): |
There was a problem hiding this comment.
_parse_with_cache can return a cached tree even when analyze_file(..., content=...) is called with explicit content for an on-disk file_path, which risks analyzing a stale tree if the provided content differs from the file on disk.
🤖 Was this useful? React with 👍 or 👎
| return | ||
| try: | ||
| # Force a sync operation to ensure all pending writes are flushed | ||
| client.get_collection(collection) |
There was a problem hiding this comment.
flush_upserts() calls get_collection(), but Qdrant’s wait=False semantics are “confirmed received” rather than “applied”, so this may not actually guarantee prior async upserts are committed before reads.
🤖 Was this useful? React with 👍 or 👎
| [scale * (1.0 if rnd.random() < 0.5 else -1.0) for _ in range(out_dim)] | ||
| for _ in range(in_dim) | ||
| ] | ||
| M = np.array(M_list, dtype=np.float32) |
There was a problem hiding this comment.
The NumPy path uses float32 for both the projection matrix and input vector, so results will differ from the pure-Python (float64) path despite the “reproducibility” comment; if determinism across environments matters, this could be surprising.
🤖 Was this useful? React with 👍 or 👎
Refines ASTAnalyzer to avoid returning stale trees when content is provided in-memory, by tracking content provenance and skipping cache in such cases. Enhances flush_upserts in Qdrant ingest to clarify consistency semantics and add a minimal scroll for better write visibility. Adds comprehensive tests for chunk deduplication, ingest infrastructure, and tree cache integration to ensure correctness and robustness.
Add tree cache to ASTAnalyzer and optimize vector projection
1 participant
Introduces a tree cache to ASTAnalyzer to avoid re-parsing unchanged files, improving performance for repeated analysis. Enhances vector projection in scripts/ingest/vectors.py by using numpy when available for significant speedup, with a fallback to pure Python. Updates Qdrant upsert logic to support async/sync modes via parameter or environment variable, and adds a flush_upserts utility to ensure data persistence after async upserts.