Embeddings response caching

[elroy-bot]

There should be a local cache for embeddings responses

this can cache responses to the embeddings function call in client.py

The cache should be a simple file system cache within the ELROY_HOME_DIR

it should in total not exceed a configurable amount of space (by default, say 500mb)

it should be a FIFO cache

[elroy-bot]

@claude attempt an implementation

[claude]

Claude finished @elroy-bot's task —— View job • claude/issue-477-20250826-1543 • Create PR ➔

---

Embeddings Response Caching Implementation Complete ✅

Successfully implemented a local filesystem cache for embeddings responses as requested.

Todo List:

• Read repository guidelines (CLAUDE.md) - No specific guidelines found
• Explore codebase to locate client.py and embeddings function
• Analyze current embeddings implementation
• Design cache system (file structure, FIFO, size management)
• Implement caching wrapper with configurable size limit (default 500MB)
• Integrate cache with existing embeddings function
• Create comprehensive unit tests
• Commit and push changes

Implementation Summary:

Created elroy/llm/embedding_cache.py:

• EmbeddingCache class with FIFO eviction policy
• Configurable size limit (default 500MB as requested)
• Persistent filesystem cache in ELROY_HOME_DIR/cache/embeddings/
• Thread-safe operations with proper error handling
• Cache statistics and management methods

Modified elroy/llm/client.py:

• Integrated cache into get_embedding() function at client.py:159-215
• Check cache before API calls (cache hits avoid API costs)
• Store results after successful API responses
• Handles edge cases like context window truncation

Added comprehensive test suite (tests/llm/test_embedding_cache.py):

• Unit tests for all cache operations
• FIFO eviction testing with size limits
• Persistence verification across cache instances
• Error handling validation

Key Features:

• FIFO Cache: Oldest entries are evicted when size limit is reached
• Persistent: Cache survives across application restarts
• Configurable: Default 500MB limit, easily adjustable
• Fast: SHA256 hashing for O(1) lookups
• Safe: Proper error handling and file cleanup

The cache will significantly reduce API costs for repeated embedding requests while staying within the configured storage limits.

---

[elroy-bot]

@claude what are the tradeoffs of doing pkl files for the cache rather than json

[claude]

Claude finished @elroy-bot's task —— View job

---

Analyzing Pickle vs JSON Tradeoffs for Embeddings Cache

Todo List:

• Examine current embeddings cache implementation
• Analyze embeddings data structure and requirements
• Compare pickle vs JSON for this specific use case
• Provide comprehensive tradeoff analysis

Pickle vs JSON for Embeddings Cache: Technical Analysis

Based on the current get_embedding() function that returns List[float], here's a comprehensive comparison:

Pickle (.pkl) Advantages:

• Performance: ~3-5x faster serialization/deserialization for numeric data
• Native Python types: Direct support for List[float] without conversion
• Precision: Maintains exact float precision (no rounding errors)
• Compact size: More efficient binary representation (~20-30% smaller files)
• Complex data: Can handle any Python object (future-proof for metadata)

Pickle Disadvantages:

• Security risk: Can execute arbitrary code during deserialization
• Python-only: Not readable by other languages/tools
• Version compatibility: Can break between Python versions
• Debugging: Binary format, can't inspect cache contents manually
• Corruption: Binary corruption can make entire file unreadable

JSON (.json) Advantages:

• Security: Text-based, no code execution risk
• Debugging: Human-readable, easy to inspect cache contents
• Portability: Language-agnostic, works with any JSON parser
• Tooling: Better ecosystem support (jq, text editors, etc.)
• Corruption resilience: Partial corruption doesn't break entire file
• Version stability: Format won't change between Python versions

JSON Disadvantages:

• Performance: Slower serialization/deserialization for large numeric arrays
• Precision loss: Float precision can be affected by JSON encoding
• File size: ~20-30% larger than pickle for numeric data
• Type conversion: Requires explicit handling of Python types

Recommendation for Embeddings Cache:

Use JSON for the following reasons:

1. Security: Embeddings cache will store external API responses - pickle poses unnecessary security risk
2. Debugging: Being able to inspect cache contents is valuable for troubleshooting
3. Performance: The performance difference is minimal for typical embedding sizes (1536 floats ≈ 12KB)
4. Reliability: JSON is more robust against corruption and version changes

The slight performance penalty is negligible compared to the network latency of embedding API calls that the cache is designed to avoid.

---

[elroy-bot]

@claude update implementation to use json

[claude]

Claude finished @elroy-bot's task —— View job • claude/issue-477-20250828-0046 • Create PR ➔

---

Embeddings Cache Implementation Updated to JSON Format ✅

Successfully converted the embeddings cache to use JSON format as requested.

Todo List:

• Examine current embedding_cache.py implementation - Found that previous implementation wasn't merged
• Create EmbeddingCache class using JSON serialization
• Integrate cache into get_embedding function in client.py
• Create unit tests for JSON format
• Test the implementation
• Commit and push changes

Implementation Summary:

Created elroy/llm/embedding_cache.py:

• EmbeddingCache class with JSON-based serialization (no pickle)
• FIFO eviction policy with configurable size limit (default 500MB)
• Persistent filesystem cache in ELROY_HOME_DIR/cache/embeddings/
• Thread-safe operations with proper error handling
• Human-readable JSON format for debugging and portability

Modified elroy/llm/client.py:

• Integrated cache into get_embedding() function at client.py:159-213
• Automatic cache checking before API calls (cache hits avoid costs)
• Caches successful API responses using JSON format
• Handles context window truncation scenarios correctly

Added comprehensive test suite (tests/llm/test_embedding_cache.py):

• Unit tests for all cache operations using JSON format
• FIFO eviction testing with size limits
• JSON format validation and persistence verification
• Error handling for corrupted JSON files
• Thread safety and large embedding support

Key Advantages of JSON over Pickle:

• Security: No code execution risk during deserialization
• Debugging: Human-readable format for inspecting cache contents
• Portability: Language-agnostic, works with any JSON parser
• Corruption resilience: Partial corruption doesn't break entire cache
• Version stability: Format won't change between Python versions

The cache maintains exact float precision and handles all embedding sizes efficiently while providing better security and debuggability than the previous pickle-based approach.

---