Conversation
BREAKING CHANGE: The `domain` module has been renamed to `document` and error types are now in their own `error` module. All imports using `vectorless::domain::*` should be updated to `vectorless::document::*`. The `Error` and `Result` types are now accessible via `vectorless::error::*`. - Rename `src/domain/mod.rs` to `src/document/mod.rs` - Move `StructureNode` and `DocumentStructure` types to new `src/document/structure.rs` module - Remove error types from document module and create separate error module - Update all imports across the codebase to reflect new module structure - Move token estimation utilities to `util` module - Update example files to use new import paths
…rarchy Add extensive error variants categorized by domain including document & parsing, index, retrieval, LLM, storage, serialization, configuration, and validation errors. Include helper methods for error categorization and context addition with retryable, not-found, timeout, and configuration error checks. Add comprehensive tests for error functionality. refactor(index): move configuration types to dedicated module Relocate IndexMode, OptimizationConfig, ThinningConfig, and PipelineOptions to src/index/config.rs to improve module organization and separation of concerns. feat(storage): add document cache with LRU eviction policy Implement thread-safe LRU cache for documents using interior mutability via Mutex. Provide cache statistics, utilization tracking, and proper error handling for concurrent access scenarios.
…ites Add comprehensive storage enhancements including: - File locking mechanism for multi-process safety using platform-specific implementations (flock on Unix, LockFileEx on Windows) - SHA-256 checksum verification for data integrity - Atomic writes using temp file + rename pattern to prevent corruption - New CompressionConfig and CompressionAlgorithm for optional compression - Enhanced StorageConfig with cache_size, atomic_writes, file_lock, checksum_enabled, and compression settings - Persistence options for configurable save/load behavior - Updated error types including ChecksumMismatch, WorkspaceLocked, and VersionMismatch - Test improvements using temporary directories and WorkspaceOptions New dependencies added: sha2 for checksums, libc for Unix file locking. BREAKING CHANGE: Storage format now includes checksums and version headers.
- Introduce AsyncWorkspace with RwLock-based concurrency support - Add StorageBackend trait with FileBackend and MemoryBackend implementations - Implement file-based storage with atomic writes and directory caching - Add in-memory backend for testing purposes - Include comprehensive async workspace functionality with caching - Update Cargo.toml with flate2 compression dependency - Export AsyncWorkspace in main library module
Remove unused mut keywords from inner variable bindings in AsyncWorkspace methods since the variables are not being mutated. feat(storage): add unsafe_code allow attribute to lock module Add unsafe_code allow attribute to the lock module along with updated documentation comments to clarify the need for unsafe FFI calls in file locking implementation.
- Add async workspace usage example demonstrating concurrent document access and async LRU cache operations - Add custom storage backend example showing how to implement StorageBackend trait with logging memory backend implementation - Add compression example demonstrating GzipCodec and IdentityCodec for compressed and uncompressed storage operations - Add version migration example showcasing migration system for upgrading data formats between versions with multi-step migration support - Add basic workspace usage example covering core storage API including document creation, loading with LRU cache, listing and removal operations These examples provide practical demonstrations of the storage module capabilities and serve as reference implementations for common use cases.
- Remove outdated examples table that listed all example files with descriptions - Simplify examples section to reference examples directory directly - Clean up formatting and reduce verbosity in documentation
- Update package version in Cargo.toml from 0.1.12 to 0.1.13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.