Conversation
Add multiple binary files containing processed document data including: - graph.bin with nodes and edges metadata - individual document binary files (doc_*.bin) with structured content - meta.bin with document metadata information These files represent sample documents with hierarchical tree structures for distributed system architecture documentation.
…uild - Change doc_key function to return id directly instead of prefixed format - Update rebuild_meta_index to filter out reserved keys instead of using prefix matching - Add reserved keys array ["meta", "_graph"] to exclude from document processing
- Add reasoning_index field to IndexedDocument struct to store pre-computed reasoning index for retrieval acceleration - Initialize reasoning_index as None in IndexedDocument default - Pass reasoning_index from indexing result to document and persist it perf(summary): improve token estimation and adjust shortcut threshold - Add token generation tracking when summaries are created - Reduce shortcut_threshold from 200 to 50 to enable faster processing for smaller documents
…tform architecture Complete rewrite of the example content to provide comprehensive documentation for the distributed data processing platform architecture. The new content covers system architecture, ingestion layer, processing engine, storage layer with tiering mechanisms, and query serving layer. Also includes deployment and operations details with monitoring and tracing setup.
- Add comprehensive documentation for the distributed data processing platform architecture - Include details about layered system architecture with clear separation of concerns between ingestion, processing, storage, and serving layers - Document microservices-based approach with independent scaling capabilities for cost-effective resource utilization - Cover real-time streaming and batch processing paradigms support - Add information about deployment and operations on Kubernetes - Include monitoring and observability details with Prometheus, Grafana, and OpenTelemetry
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.