diff --git a/rust/examples/index_single.rs b/rust/examples/index_single.rs index 8218cc00..3a5632f0 100644 --- a/rust/examples/index_single.rs +++ b/rust/examples/index_single.rs @@ -20,33 +20,43 @@ async fn main() -> vectorless::Result<()> { .await .map_err(|e| vectorless::Error::Config(e.to_string()))?; - let content = r#"# Project Overview + let content = r#"# Distributed Data Processing Platform ## Introduction -This document describes the architecture of a distributed system -designed for high-throughput data processing. +This document provides a comprehensive overview of the distributed data processing platform architecture. The system is designed to handle petabyte-scale data workloads with sub-second query latency, supporting both real-time streaming and batch processing paradigms. The architecture follows a microservices-based approach with independent scaling capabilities for each component, enabling cost-effective resource utilization across varying workload patterns. -## Components -### API Gateway +## System Architecture -Handles authentication, rate limiting, and request routing. -Supports both REST and gRPC protocols. +The platform follows a layered architecture pattern with clear separation of concerns between ingestion, processing, storage, and serving layers. Each layer can be independently deployed, scaled, and upgraded without affecting other layers, following the principle of bounded contexts from domain-driven design. Inter-layer communication uses a combination of asynchronous message passing for data flow and synchronous gRPC calls for control plane operations. -### Worker Pool +### Ingestion Layer -Processes tasks from the message queue. Each worker handles -one task at a time with configurable timeout. +The ingestion layer serves as the entry point for all data entering the platform. It supports multiple protocols including HTTP REST, gRPC, Apache Kafka, and AWS Kinesis. The layer is responsible for data validation, schema enforcement, initial transformation, and routing to downstream processing pipelines. Built on a reactive architecture using backpressure-aware operators, the ingestion layer gracefully handles burst traffic patterns without overwhelming downstream services. -## Performance -Under load testing, the system achieves 50k requests/second -with p99 latency under 200ms. +### Processing Engine -## Conclusion +The processing engine is the core computational component of the platform, responsible for transforming, enriching, aggregating, and analyzing ingested data. It supports both stream processing for real-time analytics and batch processing for historical analysis. The engine is built on a custom execution framework that optimizes query plans based on data statistics and available compute resources. -The modular design allows independent scaling of each component. +### Storage Layer + +The storage layer provides a unified abstraction over multiple storage backends, each optimized for different access patterns. The hot tier uses an in-memory columnar cache for frequently accessed dimensions and recent fact data, providing microsecond-level access latency. The warm tier uses a distributed key-value store backed by NVMe SSDs for data accessed within the past 30 days. The cold tier uses object storage with Parquet file format for historical data, achieving cost efficiency at the expense of higher access latency. + +Data is automatically tiered based on configurable policies that consider access frequency, data age, and query patterns. The tiering engine runs as a background service that continuously monitors access patterns and migrates data between tiers. Metadata about data placement is maintained in a distributed metadata service built on etcd, which provides consistent reads and writes with linearizable semantics. + +### Query Serving Layer + +The query serving layer provides the external-facing API for executing analytical queries against the processed data. It supports SQL queries via a PostgreSQL-compatible wire protocol, making it accessible to a wide range of BI tools and existing applications without requiring driver changes. The query router analyzes incoming queries and determines the optimal execution strategy, considering which storage tiers contain the relevant data and whether partial results can be served from cached aggregations. + +Query results are optionally materialized in a result cache that uses a time-to-live (TTL) policy combined with lazy invalidation based on upstream data freshness markers. The cache achieves a hit rate of approximately 85% for dashboard workloads, significantly reducing the computational load on the processing engine for repetitive query patterns. + +## Deployment and Operations + +The platform is deployed on Kubernetes with Helm charts that encapsulate all deployment configurations, resource limits, and scaling policies. Each microservice is packaged as a container image with multi-stage builds that minimize image size and attack surface. The CI/CD pipeline uses a GitOps workflow with ArgoCD, ensuring that all changes to production are auditable, reproducible, and reversible. + +Monitoring is implemented using a Prometheus and Grafana stack, with custom metrics exported by each service using a shared instrumentation library. Key performance indicators including query latency percentiles, ingestion throughput, processing lag, and error rates are tracked on operational dashboards with automated alerting through PagerDuty integration. Distributed tracing using OpenTelemetry provides end-to-end visibility into request flows across microservices, enabling rapid diagnosis of performance anomalies and error root causes. "#; // Index from content string diff --git a/rust/src/client/indexer.rs b/rust/src/client/indexer.rs index 373be62c..e746020e 100644 --- a/rust/src/client/indexer.rs +++ b/rust/src/client/indexer.rs @@ -319,6 +319,8 @@ impl IndexerClient { .with_tree(tree) .with_metrics(result.metrics); + doc.reasoning_index = result.reasoning_index; + if let Some(p) = path { doc = doc.with_source_path(p); } @@ -444,6 +446,7 @@ impl IndexerClient { persisted.add_page(page.page, &page.content); } + persisted.reasoning_index = doc.reasoning_index; persisted.meta.update_processing_stats(node_count, summary_tokens, duration_ms); persisted diff --git a/rust/src/client/types.rs b/rust/src/client/types.rs index 61201a25..079e5cfe 100644 --- a/rust/src/client/types.rs +++ b/rust/src/client/types.rs @@ -48,6 +48,9 @@ pub struct IndexedDocument { /// Indexing pipeline metrics. pub metrics: Option, + + /// Pre-computed reasoning index for retrieval acceleration. + pub reasoning_index: Option, } impl IndexedDocument { @@ -64,6 +67,7 @@ impl IndexedDocument { tree: None, pages: Vec::new(), metrics: None, + reasoning_index: None, } } diff --git a/rust/src/index/stages/enhance.rs b/rust/src/index/stages/enhance.rs index 3dc0ad81..452089c0 100644 --- a/rust/src/index/stages/enhance.rs +++ b/rust/src/index/stages/enhance.rs @@ -289,6 +289,9 @@ impl IndexStage for EnhanceStage { if summary.is_empty() { failed += 1; } else { + ctx.metrics.add_tokens_generated( + crate::utils::estimate_tokens(&summary), + ); tree.set_summary(node_id, &summary); generated += 1; ctx.metrics.increment_summaries(); diff --git a/rust/src/index/summary/strategy.rs b/rust/src/index/summary/strategy.rs index faa024bb..1e48b229 100644 --- a/rust/src/index/summary/strategy.rs +++ b/rust/src/index/summary/strategy.rs @@ -34,7 +34,7 @@ impl Default for SummaryStrategyConfig { max_tokens: 200, min_content_tokens: 50, persist_lazy: false, - shortcut_threshold: 200, + shortcut_threshold: 50, } } } diff --git a/rust/src/llm/executor.rs b/rust/src/llm/executor.rs index ddfd15ac..620f111a 100644 --- a/rust/src/llm/executor.rs +++ b/rust/src/llm/executor.rs @@ -376,12 +376,24 @@ impl LlmExecutor { let request = request.map_err(|e| LlmError::Request(format!("Failed to build request: {}", e)))?; - debug!("Sending LLM request to {} with model {}", endpoint, model); - + info!( + "LLM request → endpoint: {}, model: {}, system: {} chars, user: {} chars", + endpoint, + model, + system.len(), + truncated.len() + ); + + let request_start = std::time::Instant::now(); let response = client.chat().create(request).await.map_err(|e| { let msg = e.to_string(); LlmError::from_api_message(&msg) })?; + let request_elapsed = request_start.elapsed(); + + let usage = response.usage.as_ref(); + let prompt_tokens = usage.map(|u| u.prompt_tokens).unwrap_or(0); + let completion_tokens = usage.map(|u| u.completion_tokens).unwrap_or(0); let content = response .choices @@ -389,7 +401,13 @@ impl LlmExecutor { .and_then(|choice| choice.message.content.clone()) .ok_or(LlmError::NoContent)?; - debug!("LLM response length: {} chars", content.len()); + info!( + "LLM response ← {}ms, tokens: {} prompt + {} completion, content: {} chars", + request_elapsed.as_millis(), + prompt_tokens, + completion_tokens, + content.len() + ); Ok(content) } diff --git a/rust/src/storage/workspace.rs b/rust/src/storage/workspace.rs index 476df6fe..045cf1e6 100644 --- a/rust/src/storage/workspace.rs +++ b/rust/src/storage/workspace.rs @@ -482,7 +482,7 @@ impl Workspace { /// Get the storage key for a document. fn doc_key(id: &str) -> String { - format!("doc:{}", id) + id.to_string() } /// Load the meta index from backend. @@ -516,7 +516,11 @@ impl Workspace { /// Rebuild the meta index from existing documents. fn rebuild_meta_index(inner: &mut WorkspaceInner) -> Result<()> { let keys = inner.backend.keys()?; - let doc_keys: Vec<_> = keys.iter().filter(|k| k.starts_with("doc:")).collect(); + let reserved = ["meta", "_graph"]; + let doc_keys: Vec<_> = keys + .iter() + .filter(|k| !reserved.contains(&k.as_str())) + .collect(); for key in doc_keys { if let Some(bytes) = inner.backend.get(key)? { diff --git a/samples/1938cb46-4085-4a70-b9e6-70b97d3c8ba9.bin b/samples/1938cb46-4085-4a70-b9e6-70b97d3c8ba9.bin new file mode 100644 index 00000000..31bd59ff --- /dev/null +++ b/samples/1938cb46-4085-4a70-b9e6-70b97d3c8ba9.bin @@ -0,0 +1 @@ +{"version":1,"checksum":"b27a76f02225295a9dc19b2de9458d1200070919b226a005cd0b59e2c592584c","payload":{"meta":{"id":"1938cb46-4085-4a70-b9e6-70b97d3c8ba9","name":"","format":"md","source_path":"","description":"","page_count":null,"line_count":null,"created_at":"2026-04-12T04:46:05.866270414Z","modified_at":"2026-04-12T04:46:05.866414689Z","logic_fingerprint":"b25J69t2pTTx/z0WFznfGw==","processing_version":0,"node_count":9,"total_summary_tokens":444,"processing_duration_ms":69435},"tree":{"arena":{"nodes":[{"parent":null,"previous_sibling":null,"next_sibling":null,"first_child":{"index1":2,"stamp":0},"last_child":{"index1":2,"stamp":0},"stamp":0,"data":{"Data":{"title":"","structure":"","content":"","summary":"","depth":0,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0001","physical_index":null,"token_count":null,"references":[]}}},{"parent":{"index1":1,"stamp":0},"previous_sibling":null,"next_sibling":null,"first_child":{"index1":3,"stamp":0},"last_child":{"index1":9,"stamp":0},"stamp":0,"data":{"Data":{"title":"Distributed Data Processing Platform","structure":"1","content":"","summary":"","depth":1,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0002","physical_index":null,"token_count":null,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":null,"next_sibling":{"index1":4,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Introduction","structure":"1.1","content":"This document provides a comprehensive overview of the distributed data processing platform architecture. The system is designed to handle petabyte-scale data workloads with sub-second query latency, supporting both real-time streaming and batch processing paradigms. The architecture follows a microservices-based approach with independent scaling capabilities for each component, enabling cost-effective resource utilization across varying workload patterns.","summary":"This document outlines a distributed data processing platform built to manage petabyte-scale workloads with sub-second latency. By utilizing a microservices-based architecture, the system supports both batch and real-time streaming while allowing independent component scaling for cost-effective resource utilization.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0003","physical_index":null,"token_count":70,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":{"index1":3,"stamp":0},"next_sibling":{"index1":9,"stamp":0},"first_child":{"index1":5,"stamp":0},"last_child":{"index1":8,"stamp":0},"stamp":0,"data":{"Data":{"title":"System Architecture","structure":"1.2","content":"The platform follows a layered architecture pattern with clear separation of concerns between ingestion, processing, storage, and serving layers. Each layer can be independently deployed, scaled, and upgraded without affecting other layers, following the principle of bounded contexts from domain-driven design. Inter-layer communication uses a combination of asynchronous message passing for data flow and synchronous gRPC calls for control plane operations.","summary":"This section details the platform's layered system architecture, encompassing the ingestion, processing, storage, and serving layers. It explains the separation of concerns and independent scalability based on domain-driven design principles. Furthermore, it covers the inter-layer communication mechanisms, including asynchronous message passing and synchronous gRPC calls.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0004","physical_index":null,"token_count":73,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":null,"next_sibling":{"index1":6,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Ingestion Layer","structure":"1.2.1","content":"The ingestion layer serves as the entry point for all data entering the platform. It supports multiple protocols including HTTP REST, gRPC, Apache Kafka, and AWS Kinesis. The layer is responsible for data validation, schema enforcement, initial transformation, and routing to downstream processing pipelines. Built on a reactive architecture using backpressure-aware operators, the ingestion layer gracefully handles burst traffic patterns without overwhelming downstream services.","summary":"The ingestion layer acts as the primary entry point for data entering the platform, supporting diverse protocols such as HTTP REST, gRPC, Kafka, and Kinesis. It handles critical preliminary tasks including data validation, schema enforcement, transformation, and routing. Additionally, its reactive, backpressure-aware architecture ensures that burst traffic is managed smoothly without overwhelming downstream services.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0005","physical_index":null,"token_count":79,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":5,"stamp":0},"next_sibling":{"index1":7,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Processing Engine","structure":"1.2.2","content":"The processing engine is the core computational component of the platform, responsible for transforming, enriching, aggregating, and analyzing ingested data. It supports both stream processing for real-time analytics and batch processing for historical analysis. The engine is built on a custom execution framework that optimizes query plans based on data statistics and available compute resources.","summary":"The processing engine serves as the platform's core computational component, responsible for transforming and analyzing ingested data through both real-time stream and historical batch processing. It operates on a custom execution framework that dynamically optimizes query plans based on data statistics and available compute resources.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0006","physical_index":null,"token_count":67,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":6,"stamp":0},"next_sibling":{"index1":8,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Storage Layer","structure":"1.2.3","content":"The storage layer provides a unified abstraction over multiple storage backends, each optimized for different access patterns. The hot tier uses an in-memory columnar cache for frequently accessed dimensions and recent fact data, providing microsecond-level access latency. The warm tier uses a distributed key-value store backed by NVMe SSDs for data accessed within the past 30 days. The cold tier uses object storage with Parquet file format for historical data, achieving cost efficiency at the expense of higher access latency.Data is automatically tiered based on configurable policies that consider access frequency, data age, and query patterns. The tiering engine runs as a background service that continuously monitors access patterns and migrates data between tiers. Metadata about data placement is maintained in a distributed metadata service built on etcd, which provides consistent reads and writes with linearizable semantics.","summary":"The storage layer uses a multi-tiered architecture—spanning an in-memory cache, NVMe SSDs, and cost-efficient object storage—to optimize performance for different data access patterns. A background tiering engine automatically migrates data between these hot, warm, and cold tiers based on access frequency, data age, and query patterns. Data placement metadata is consistently tracked and managed across these backends using a distributed etcd-based service.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0007","physical_index":null,"token_count":165,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":7,"stamp":0},"next_sibling":null,"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Query Serving Layer","structure":"1.2.4","content":"The query serving layer provides the external-facing API for executing analytical queries against the processed data. It supports SQL queries via a PostgreSQL-compatible wire protocol, making it accessible to a wide range of BI tools and existing applications without requiring driver changes. The query router analyzes incoming queries and determines the optimal execution strategy, considering which storage tiers contain the relevant data and whether partial results can be served from cached aggregations.Query results are optionally materialized in a result cache that uses a time-to-live (TTL) policy combined with lazy invalidation based on upstream data freshness markers. The cache achieves a hit rate of approximately 85% for dashboard workloads, significantly reducing the computational load on the processing engine for repetitive query patterns.","summary":"The query serving layer acts as an external API for executing analytical SQL queries through a PostgreSQL-compatible protocol. It utilizes a query router to optimize execution strategies by analyzing storage tiers and leveraging cached aggregations. Additionally, a dedicated result cache achieves an 85% hit rate for dashboard workloads, significantly reducing the computational load on the processing engine.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0008","physical_index":null,"token_count":142,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":{"index1":4,"stamp":0},"next_sibling":null,"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Deployment and Operations","structure":"1.3","content":"The platform is deployed on Kubernetes with Helm charts that encapsulate all deployment configurations, resource limits, and scaling policies. Each microservice is packaged as a container image with multi-stage builds that minimize image size and attack surface. The CI/CD pipeline uses a GitOps workflow with ArgoCD, ensuring that all changes to production are auditable, reproducible, and reversible.Monitoring is implemented using a Prometheus and Grafana stack, with custom metrics exported by each service using a shared instrumentation library. Key performance indicators including query latency percentiles, ingestion throughput, processing lag, and error rates are tracked on operational dashboards with automated alerting through PagerDuty integration. Distributed tracing using OpenTelemetry provides end-to-end visibility into request flows across microservices, enabling rapid diagnosis of performance anomalies and error root causes.","summary":"The platform is deployed on Kubernetes using containerized microservices and managed through an automated GitOps CI/CD pipeline with ArgoCD. Comprehensive system observability is maintained using a Prometheus and Grafana stack for metrics, alongside OpenTelemetry for distributed tracing, which collectively enable automated alerting and rapid troubleshooting.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0009","physical_index":null,"token_count":163,"references":[]}}}],"first_free_slot":null,"last_free_slot":null},"root_id":{"index1":1,"stamp":0}},"pages":[],"reasoning_index":{"topic_paths":{"migrates":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"layered":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"encompassing":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"ssds":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"ci":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"architecture":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":3,"stamp":0},"weight":0.42857146,"depth":2},{"node_id":{"index1":5,"stamp":0},"weight":0.42857146,"depth":3},{"node_id":{"index1":7,"stamp":0},"weight":0.42857146,"depth":3}],"burst":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"available":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"significantly":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"layers":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"transforming":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"scalability":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"prometheus":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"critical":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"http":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"section":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"consistently":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"point":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"observability":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"age":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"compatible":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"ingestion":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.42857146,"depth":2}],"kubernetes":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"reactive":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"document":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"principles":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"kafka":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"tiered":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"alerting":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"layer":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.42857146,"depth":2}],"efficient":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"passing":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"built":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"workloads":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"details":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"result":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"aggregations":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"routing":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"diverse":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"ingested":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"external":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"manage":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"resources":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"domain":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"outlines":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"cd":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"responsible":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"automated":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"hit":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"hot":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"without":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"ensures":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"preliminary":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"metrics":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"both":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"queries":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"object":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"validation":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"real":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"introduction":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"execution":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"deployment":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"platform":[{"node_id":{"index1":2,"stamp":0},"weight":1.0,"depth":1},{"node_id":{"index1":3,"stamp":0},"weight":0.75,"depth":2},{"node_id":{"index1":5,"stamp":0},"weight":0.75,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.75,"depth":2},{"node_id":{"index1":6,"stamp":0},"weight":0.75,"depth":3},{"node_id":{"index1":9,"stamp":0},"weight":0.75,"depth":2}],"grpc":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"85":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"supporting":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"sql":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"entering":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"optimizes":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"analytical":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"microservices":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"across":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"system":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":9,"stamp":0},"weight":0.42857146,"depth":2},{"node_id":{"index1":3,"stamp":0},"weight":0.42857146,"depth":2}],"grafana":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"plans":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"rest":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"utilization":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"multi":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"postgresql":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"rate":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"transformation":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"strategies":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"background":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"message":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"protocol":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"enable":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"component":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"tiering":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"second":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"dashboard":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"independent":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"downstream":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"primary":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"using":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":7,"stamp":0},"weight":0.5,"depth":3}],"cost":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"based":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":0.5,"depth":2},{"node_id":{"index1":6,"stamp":0},"weight":0.5,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.5,"depth":2}],"scale":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"uses":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"etcd":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"driven":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"backpressure":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"explains":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"streaming":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"schema":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"executing":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"operations":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"automatically":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"cached":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"collectively":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"aware":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"design":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"overwhelming":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"opentelemetry":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"comprehensive":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"inter":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"protocols":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"achieves":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"cache":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"alongside":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"data":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":5,"stamp":0},"weight":0.5,"depth":3},{"node_id":{"index1":6,"stamp":0},"weight":0.5,"depth":3},{"node_id":{"index1":2,"stamp":0},"weight":0.33333334,"depth":1},{"node_id":{"index1":3,"stamp":0},"weight":0.25,"depth":2}],"metadata":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"nvme":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"memory":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"troubleshooting":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"query":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":7,"stamp":0},"weight":0.3,"depth":3},{"node_id":{"index1":6,"stamp":0},"weight":0.3,"depth":3}],"tiers":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"frequency":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"computational":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"compute":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"dedicated":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"furthermore":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"statistics":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"operates":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"managed":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"leveraging":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"synchronous":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"tracked":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"containerized":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"mechanisms":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"traffic":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"dynamically":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"latency":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"tasks":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"router":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"covers":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"custom":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"additionally":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"service":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"services":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"serves":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"tracing":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"warm":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"different":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"utilizing":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"cold":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"supports":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"stack":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"rapid":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"deployed":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"core":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"allowing":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"resource":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"reducing":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"backends":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"gitops":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"patterns":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"communication":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"maintained":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"effective":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"acts":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"petabyte":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"load":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"handles":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"historical":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"scaling":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"separation":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"stream":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"asynchronous":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"access":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"framework":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"processing":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":2,"stamp":0},"weight":0.4,"depth":1},{"node_id":{"index1":8,"stamp":0},"weight":0.3,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":0.3,"depth":2},{"node_id":{"index1":4,"stamp":0},"weight":0.3,"depth":2}],"pipeline":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"batch":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"including":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"serving":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.42857146,"depth":2}],"spanning":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"performance":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"storage":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":0.3,"depth":3},{"node_id":{"index1":4,"stamp":0},"weight":0.3,"depth":2}],"time":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2},{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"analyzing":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3}],"argocd":[{"node_id":{"index1":9,"stamp":0},"weight":1.0,"depth":2}],"enforcement":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"placement":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3}],"utilizes":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"entry":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"concerns":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"api":[{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"optimize":[{"node_id":{"index1":7,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":1.0,"depth":3}],"calls":[{"node_id":{"index1":4,"stamp":0},"weight":1.0,"depth":2}],"smoothly":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}],"sub":[{"node_id":{"index1":3,"stamp":0},"weight":1.0,"depth":2}],"engine":[{"node_id":{"index1":6,"stamp":0},"weight":1.0,"depth":3},{"node_id":{"index1":7,"stamp":0},"weight":0.42857146,"depth":3},{"node_id":{"index1":8,"stamp":0},"weight":0.42857146,"depth":3}],"distributed":[{"node_id":{"index1":2,"stamp":0},"weight":1.0,"depth":1},{"node_id":{"index1":3,"stamp":0},"weight":0.75,"depth":2},{"node_id":{"index1":7,"stamp":0},"weight":0.75,"depth":3},{"node_id":{"index1":9,"stamp":0},"weight":0.75,"depth":2}],"kinesis":[{"node_id":{"index1":5,"stamp":0},"weight":1.0,"depth":3}]},"summary_shortcut":{"root_node":{"index1":1,"stamp":0},"section_summaries":[{"node_id":{"index1":2,"stamp":0},"title":"Distributed Data Processing Platform","summary":"","depth":1}],"document_summary":""},"hot_nodes":{},"section_map":{"1":{"index1":2,"stamp":0},"distributed data processing platform":{"index1":2,"stamp":0}},"config_hash":0}}} \ No newline at end of file diff --git a/samples/5dfb586a-9e7a-4087-ad8f-24eb09281269.bin b/samples/5dfb586a-9e7a-4087-ad8f-24eb09281269.bin new file mode 100644 index 00000000..79f6334c --- /dev/null +++ b/samples/5dfb586a-9e7a-4087-ad8f-24eb09281269.bin @@ -0,0 +1 @@ +{"version":1,"checksum":"6206fa81bcf6e44bec5cfb6d37648200284f7ad8e0bb1583e4d771a31fde9164","payload":{"meta":{"id":"5dfb586a-9e7a-4087-ad8f-24eb09281269","name":"","format":"md","source_path":"","description":"","page_count":null,"line_count":null,"created_at":"2026-04-12T04:20:11.338077430Z","modified_at":"2026-04-12T04:20:11.338222591Z","logic_fingerprint":"b25J69t2pTTx/z0WFznfGw==","processing_version":0,"node_count":9,"total_summary_tokens":0,"processing_duration_ms":31021},"tree":{"arena":{"nodes":[{"parent":null,"previous_sibling":null,"next_sibling":null,"first_child":{"index1":2,"stamp":0},"last_child":{"index1":2,"stamp":0},"stamp":0,"data":{"Data":{"title":"","structure":"","content":"","summary":"","depth":0,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0001","physical_index":null,"token_count":null,"references":[]}}},{"parent":{"index1":1,"stamp":0},"previous_sibling":null,"next_sibling":null,"first_child":{"index1":3,"stamp":0},"last_child":{"index1":9,"stamp":0},"stamp":0,"data":{"Data":{"title":"Distributed Data Processing Platform","structure":"1","content":"","summary":"","depth":1,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0002","physical_index":null,"token_count":null,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":null,"next_sibling":{"index1":4,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Introduction","structure":"1.1","content":"This document provides a comprehensive overview of the distributed data processing platform architecture. The system is designed to handle petabyte-scale data workloads with sub-second query latency, supporting both real-time streaming and batch processing paradigms. The architecture follows a microservices-based approach with independent scaling capabilities for each component, enabling cost-effective resource utilization across varying workload patterns.","summary":"This document outlines a distributed data processing platform designed to handle petabyte-scale workloads with sub-second latency for both streaming and batch processing. It utilizes a microservices-based architecture that enables independent component scaling to ensure cost-effective resource utilization.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0003","physical_index":null,"token_count":70,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":{"index1":3,"stamp":0},"next_sibling":{"index1":9,"stamp":0},"first_child":{"index1":5,"stamp":0},"last_child":{"index1":8,"stamp":0},"stamp":0,"data":{"Data":{"title":"System Architecture","structure":"1.2","content":"The platform follows a layered architecture pattern with clear separation of concerns between ingestion, processing, storage, and serving layers. Each layer can be independently deployed, scaled, and upgraded without affecting other layers, following the principle of bounded contexts from domain-driven design. Inter-layer communication uses a combination of asynchronous message passing for data flow and synchronous gRPC calls for control plane operations.","summary":"This section provides an overview of the platform's layered system architecture, specifically detailing the ingestion, processing, storage, and serving components. It also covers deployment and scaling strategies based on domain-driven design, as well as the inter-layer communication protocols utilizing asynchronous messaging and synchronous gRPC.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0004","physical_index":null,"token_count":73,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":null,"next_sibling":{"index1":6,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Ingestion Layer","structure":"1.2.1","content":"The ingestion layer serves as the entry point for all data entering the platform. It supports multiple protocols including HTTP REST, gRPC, Apache Kafka, and AWS Kinesis. The layer is responsible for data validation, schema enforcement, initial transformation, and routing to downstream processing pipelines. Built on a reactive architecture using backpressure-aware operators, the ingestion layer gracefully handles burst traffic patterns without overwhelming downstream services.","summary":"The ingestion layer serves as the primary entry point for platform data, supporting multiple protocols such as HTTP REST, gRPC, Apache Kafka, and AWS Kinesis. It is responsible for essential early-stage tasks including data validation, schema enforcement, transformation, and routing to downstream pipelines. Additionally, its reactive, backpressure-aware architecture enables it to gracefully handle sudden traffic bursts without overwhelming downstream services.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0005","physical_index":null,"token_count":79,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":5,"stamp":0},"next_sibling":{"index1":7,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Processing Engine","structure":"1.2.2","content":"The processing engine is the core computational component of the platform, responsible for transforming, enriching, aggregating, and analyzing ingested data. It supports both stream processing for real-time analytics and batch processing for historical analysis. The engine is built on a custom execution framework that optimizes query plans based on data statistics and available compute resources.","summary":"The processing engine serves as the platform's core computational component, responsible for transforming, enriching, and analyzing ingested data. It supports both real-time stream processing and historical batch processing, utilizing a custom execution framework to dynamically optimize query plans based on data statistics and available compute resources.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0006","physical_index":null,"token_count":67,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":6,"stamp":0},"next_sibling":{"index1":8,"stamp":0},"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Storage Layer","structure":"1.2.3","content":"The storage layer provides a unified abstraction over multiple storage backends, each optimized for different access patterns. The hot tier uses an in-memory columnar cache for frequently accessed dimensions and recent fact data, providing microsecond-level access latency. The warm tier uses a distributed key-value store backed by NVMe SSDs for data accessed within the past 30 days. The cold tier uses object storage with Parquet file format for historical data, achieving cost efficiency at the expense of higher access latency.Data is automatically tiered based on configurable policies that consider access frequency, data age, and query patterns. The tiering engine runs as a background service that continuously monitors access patterns and migrates data between tiers. Metadata about data placement is maintained in a distributed metadata service built on etcd, which provides consistent reads and writes with linearizable semantics.","summary":"The storage layer is a multi-tiered system using in-memory caching (hot), NVMe SSDs (warm), and object storage (cold) to optimize for both performance and cost. A background tiering engine automatically migrates data between these tiers based on configurable policies, data age, and access patterns. The system relies on an etcd-backed distributed metadata service to consistently track data placement across all tiers.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0007","physical_index":null,"token_count":165,"references":[]}}},{"parent":{"index1":4,"stamp":0},"previous_sibling":{"index1":7,"stamp":0},"next_sibling":null,"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Query Serving Layer","structure":"1.2.4","content":"The query serving layer provides the external-facing API for executing analytical queries against the processed data. It supports SQL queries via a PostgreSQL-compatible wire protocol, making it accessible to a wide range of BI tools and existing applications without requiring driver changes. The query router analyzes incoming queries and determines the optimal execution strategy, considering which storage tiers contain the relevant data and whether partial results can be served from cached aggregations.Query results are optionally materialized in a result cache that uses a time-to-live (TTL) policy combined with lazy invalidation based on upstream data freshness markers. The cache achieves a hit rate of approximately 85% for dashboard workloads, significantly reducing the computational load on the processing engine for repetitive query patterns.","summary":"The query serving layer provides a PostgreSQL-compatible API that enables BI tools to seamlessly execute analytical queries. A query router optimizes these executions by evaluating storage tiers and available cached aggregations. To significantly reduce the computational load on the processing engine, a result cache utilizing time-to-live and lazy invalidation achieves an 85% hit rate for dashboard workloads.","depth":3,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0008","physical_index":null,"token_count":142,"references":[]}}},{"parent":{"index1":2,"stamp":0},"previous_sibling":{"index1":4,"stamp":0},"next_sibling":null,"first_child":null,"last_child":null,"stamp":0,"data":{"Data":{"title":"Deployment and Operations","structure":"1.3","content":"The platform is deployed on Kubernetes with Helm charts that encapsulate all deployment configurations, resource limits, and scaling policies. Each microservice is packaged as a container image with multi-stage builds that minimize image size and attack surface. The CI/CD pipeline uses a GitOps workflow with ArgoCD, ensuring that all changes to production are auditable, reproducible, and reversible.Monitoring is implemented using a Prometheus and Grafana stack, with custom metrics exported by each service using a shared instrumentation library. Key performance indicators including query latency percentiles, ingestion throughput, processing lag, and error rates are tracked on operational dashboards with automated alerting through PagerDuty integration. Distributed tracing using OpenTelemetry provides end-to-end visibility into request flows across microservices, enabling rapid diagnosis of performance anomalies and error root causes.","summary":"The platform is deployed on Kubernetes using Helm charts and containerized microservices, with deployments managed by a GitOps CI/CD pipeline via ArgoCD to ensure auditable and reversible updates. Operational observability is achieved using a Prometheus and Grafana stack for monitoring key metrics and automated alerting, combined with OpenTelemetry for distributed tracing to rapidly diagnose performance issues.","depth":2,"start_index":1,"end_index":1,"start_page":null,"end_page":null,"node_id":"0009","physical_index":null,"token_count":163,"references":[]}}}],"first_free_slot":null,"last_free_slot":null},"root_id":{"index1":1,"stamp":0}},"pages":[]}} \ No newline at end of file diff --git a/samples/_graph.bin b/samples/_graph.bin new file mode 100644 index 00000000..e1d51275 --- /dev/null +++ b/samples/_graph.bin @@ -0,0 +1 @@ +{"nodes":{"5dfb586a-9e7a-4087-ad8f-24eb09281269":{"doc_id":"5dfb586a-9e7a-4087-ad8f-24eb09281269","title":"","format":"md","top_keywords":[],"node_count":9}},"edges":{},"keyword_index":{},"metadata":{"document_count":1,"edge_count":0}} \ No newline at end of file diff --git a/samples/meta.bin b/samples/meta.bin new file mode 100644 index 00000000..a3518791 --- /dev/null +++ b/samples/meta.bin @@ -0,0 +1,23 @@ +{ + "5dfb586a-9e7a-4087-ad8f-24eb09281269": { + "id": "5dfb586a-9e7a-4087-ad8f-24eb09281269", + "doc_name": "", + "doc_description": "", + "doc_type": "md", + "path": "" + }, + "6998c9ff-ba74-4762-9f3e-c838be2f425b": { + "id": "6998c9ff-ba74-4762-9f3e-c838be2f425b", + "doc_name": "", + "doc_description": "", + "doc_type": "md", + "path": "" + }, + "1938cb46-4085-4a70-b9e6-70b97d3c8ba9": { + "id": "1938cb46-4085-4a70-b9e6-70b97d3c8ba9", + "doc_name": "", + "doc_description": "", + "doc_type": "md", + "path": "" + } +} \ No newline at end of file