Skip to content

Commit 53b4e91

Browse files
committed
feat(messagehistory): implement SemanticMessageHistory with vector search
Adds SemanticMessageHistory class for storing and retrieving LLM conversation history with vector embeddings for semantic search. This matches the Python redis-vl-python SemanticMessageHistory implementation. New classes: - SemanticMessageHistory: Main class with getRelevant() for semantic search - SemanticMessageHistorySchema: Schema with vector field for embeddings - SemanticChatMessage: Message model with vector field support Features: - Automatic message embedding using configurable vectorizer - getRelevant(prompt) for semantic similarity search on history - Configurable distance threshold for search precision - Role filtering (user, llm, system, tool) in semantic search - Fallback to recent messages when no semantic matches - Session tagging for multi-conversation support Tests: - 22 unit tests for SemanticMessageHistory - 15 integration tests with Redis Stack Documentation: - Updated message-history.adoc with full SemanticMessageHistory examples - Updated FEATURE_PARITY_REPORT.md to reflect ~95% parity
1 parent f56f762 commit 53b4e91

File tree

8 files changed

+2083
-5
lines changed

8 files changed

+2083
-5
lines changed

FEATURE_PARITY_REPORT.md

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
# RedisVL4J vs Python redis-vl-python Feature Parity Report
2+
3+
**Generated:** 2024-12-10
4+
**Python Version Reference:** redis-vl-python (latest main branch)
5+
**Java Version Reference:** RedisVL4J (current main branch)
6+
7+
---
8+
9+
## Executive Summary
10+
11+
RedisVL4J has achieved **~95% feature parity** with the Python redis-vl-python library. The core functionality (schema, index, queries, vectorizers, rerankers, caching, routing, SVS-VAMANA, semantic message history) is complete. The main gaps are in:
12+
13+
1. **Vectorizer integrations** - Missing several cloud provider integrations (use LangChain4J as workaround)
14+
2. **CLI Tools** - No command-line interface
15+
3. **Async Support** - No async variants (Java uses different concurrency models)
16+
4. **Compression utilities** - Missing CompressionAdvisor helper (low priority)
17+
18+
---
19+
20+
## Detailed Feature Comparison
21+
22+
### Legend
23+
- ✅ Fully Implemented
24+
- ⚠️ Partially Implemented
25+
- ❌ Not Implemented
26+
- N/A Not Applicable (platform-specific)
27+
28+
---
29+
30+
## 1. SCHEMA AND INDEX MANAGEMENT
31+
32+
| Feature | Python | Java | Notes |
33+
|---------|--------|------|-------|
34+
| **IndexSchema** ||| Full parity |
35+
| **StorageType (HASH/JSON)** ||| Full parity |
36+
| **Multiple prefixes** ||| Full parity |
37+
| **Stopwords configuration** ||| Full parity |
38+
| **YAML/JSON schema loading** ||| Full parity |
39+
| **Schema validation** ||| Full parity |
40+
| **IndexInfo** || ⚠️ | Basic info available |
41+
42+
### Field Types
43+
44+
| Field Type | Python | Java | Notes |
45+
|------------|--------|------|-------|
46+
| **TextField** ||| Full parity |
47+
| **TagField** ||| Full parity |
48+
| **NumericField** ||| Full parity |
49+
| **GeoField** ||| Full parity |
50+
| **FlatVectorField** ||| Full parity |
51+
| **HNSWVectorField** ||| Full parity |
52+
| **SVSVectorField** ||| Full parity (commit 379fa2b) |
53+
54+
### Vector Field Attributes
55+
56+
| Attribute | Python | Java | Notes |
57+
|-----------|--------|------|-------|
58+
| dims ||| |
59+
| distance_metric (COSINE, L2, IP) ||| |
60+
| datatype (float16, float32, float64) ||| |
61+
| datatype (bfloat16, int8, uint8) ||| Full parity (commit 379fa2b) |
62+
| HNSW: m, ef_construction, ef_runtime ||| |
63+
| HNSW: epsilon ||| Added in PR #439 |
64+
| SVS-VAMANA: compression types ||| LVQ4, LVQ4x4, LVQ4x8, LVQ8, LeanVec4x8, LeanVec8x8 |
65+
| SVS-VAMANA: graph parameters ||| graphMaxDegree, constructionWindowSize, searchWindowSize, etc. |
66+
67+
---
68+
69+
## 2. SEARCH INDEX OPERATIONS
70+
71+
| Feature | Python | Java | Notes |
72+
|---------|--------|------|-------|
73+
| **SearchIndex** ||| Full parity |
74+
| **AsyncSearchIndex** || N/A | Java uses different async patterns |
75+
| create() / drop() / exists() ||| |
76+
| addDocument() / updateDocument() ||| |
77+
| deleteDocument() ||| |
78+
| load() (batch loading) ||| |
79+
| fetch() ||| |
80+
| clear() ||| |
81+
| getInfo() / getDocumentCount() ||| |
82+
| search() / query() ||| |
83+
| paginate() ||| |
84+
| batchSearch() / batchQuery() ||| |
85+
| expireKeys() ||| |
86+
| from_existing() ||| |
87+
| listall() ||| |
88+
89+
---
90+
91+
## 3. QUERY TYPES
92+
93+
| Query Type | Python | Java | Notes |
94+
|------------|--------|------|-------|
95+
| **VectorQuery** ||| Full parity |
96+
| **VectorRangeQuery / RangeQuery** ||| Full parity |
97+
| **FilterQuery** ||| Full parity |
98+
| **TextQuery** ||| Full parity |
99+
| **CountQuery** ||| Full parity |
100+
| **HybridQuery** ||| Full parity |
101+
| **AggregationQuery** ||| Full parity |
102+
| **MultiVectorQuery** ||| Full parity |
103+
104+
### Query Features
105+
106+
| Feature | Python | Java | Notes |
107+
|---------|--------|------|-------|
108+
| Pre-filtering ||| |
109+
| Runtime params (efRuntime) ||| PR #439 |
110+
| Runtime params (epsilon) ||| PR #439 |
111+
| SVS-VAMANA runtime params ||| PR #439 (searchWindowSize, etc.) |
112+
| Return fields ||| |
113+
| Skip decode fields ||| |
114+
| Sorting (single/multi-field) ||| |
115+
| Hybrid alpha parameter ||| |
116+
| Distance normalization ||| |
117+
118+
### Filter Types
119+
120+
| Filter | Python | Java | Notes |
121+
|--------|--------|------|-------|
122+
| Tag filter ||| |
123+
| Text filter ||| |
124+
| Numeric filter (>, <, ==, >=, <=) ||| |
125+
| Geo filter (radius, box) ||| |
126+
| Wildcard ||| |
127+
| Prefix ||| |
128+
| Fuzzy ||| |
129+
| AND / OR / NOT combinators ||| |
130+
131+
---
132+
133+
## 4. VECTORIZER INTEGRATIONS
134+
135+
| Vectorizer | Python | Java | Notes |
136+
|------------|--------|------|-------|
137+
| **BaseVectorizer** ||| |
138+
| **HFTextVectorizer** (HuggingFace) || ⚠️ | Java has SentenceTransformersVectorizer |
139+
| **OpenAITextVectorizer** ||| **GAP** |
140+
| **AzureOpenAITextVectorizer** ||| **GAP** |
141+
| **CohereTextVectorizer** ||| **GAP** |
142+
| **MistralAITextVectorizer** ||| **GAP** |
143+
| **VertexAITextVectorizer** ||| **GAP** |
144+
| **BedrockTextVectorizer** ||| **GAP** |
145+
| **VoyageAITextVectorizer** ||| **GAP** |
146+
| **CustomTextVectorizer** ||| Via BaseVectorizer extension |
147+
| **LangChain4JVectorizer** | N/A || Java-specific integration |
148+
| **SentenceTransformersVectorizer** | ✅ (via HF) || ONNX-based local models |
149+
| **MockVectorizer** ||| For testing |
150+
| **vectorizer_from_dict()** ||| **GAP: Factory function** |
151+
| **EmbeddingsCache integration** ||| |
152+
153+
---
154+
155+
## 5. RERANKER IMPLEMENTATIONS
156+
157+
| Reranker | Python | Java | Notes |
158+
|----------|--------|------|-------|
159+
| **BaseReranker** ||| Full parity |
160+
| **CohereReranker** ||| Full parity |
161+
| **HFCrossEncoderReranker** ||| Full parity |
162+
| **VoyageAIReranker** ||| Full parity |
163+
164+
---
165+
166+
## 6. LLM CACHE IMPLEMENTATIONS
167+
168+
| Feature | Python | Java | Notes |
169+
|---------|--------|------|-------|
170+
| **BaseCache** ||| |
171+
| **SemanticCache** ||| Full parity |
172+
| **LangCacheSemanticCache** ||| Full parity (PR #442) |
173+
| URL percent-encoding ||| PR #442 |
174+
| Per-entry TTL ||| PR #442 |
175+
| **EmbeddingsCache** ||| Full parity |
176+
| CacheEntry / CacheHit models || ⚠️ | Basic support |
177+
| SemanticCacheIndexSchema || ⚠️ | Auto-generated |
178+
| Async cache operations || N/A | |
179+
180+
---
181+
182+
## 7. MESSAGE HISTORY / SESSION MANAGEMENT
183+
184+
| Feature | Python | Java | Notes |
185+
|---------|--------|------|-------|
186+
| **BaseMessageHistory** ||| |
187+
| **MessageHistory** (Standard) ||| Full parity |
188+
| **SemanticMessageHistory** ||| Full parity - `getRelevant()` semantic search |
189+
| Session tagging ||| |
190+
| Role filtering (user, llm, system, tool) ||| |
191+
| TTL management ||| |
192+
| Tool call storage ||| |
193+
| Metadata storage ||| |
194+
| get_recent() ||| |
195+
| get_relevant() (semantic search) ||| **GAP** |
196+
| Deprecated SessionManager aliases || N/A | Not needed in Java |
197+
198+
---
199+
200+
## 8. SEMANTIC ROUTER
201+
202+
| Feature | Python | Java | Notes |
203+
|---------|--------|------|-------|
204+
| **SemanticRouter** ||| Full parity |
205+
| **Route** ||| Full parity |
206+
| **RoutingConfig** ||| Full parity |
207+
| **RouteMatch** ||| Full parity |
208+
| Distance aggregation methods ||| |
209+
| Vectorizer integration ||| |
210+
| Auto-schema generation ||| |
211+
212+
---
213+
214+
## 9. LANGCHAIN4J INTEGRATIONS (Java-Specific)
215+
216+
| Feature | Python | Java | Notes |
217+
|---------|--------|------|-------|
218+
| **EmbeddingStore** | N/A || Java-specific |
219+
| **ContentRetriever** | N/A || Java-specific |
220+
| **DocumentStore** | N/A || Java-specific |
221+
| **ChatMemoryStore** | N/A || Java-specific |
222+
| **FilterMapper** | N/A || Java-specific |
223+
224+
---
225+
226+
## 10. CLI TOOLS
227+
228+
| Feature | Python | Java | Notes |
229+
|---------|--------|------|-------|
230+
| **rvl CLI** ||| **GAP: No CLI** |
231+
| rvl index create ||| |
232+
| rvl index delete ||| |
233+
| rvl index info ||| |
234+
| rvl stats ||| |
235+
| rvl version ||| |
236+
237+
---
238+
239+
## 11. UTILITIES
240+
241+
| Utility | Python | Java | Notes |
242+
|---------|--------|------|-------|
243+
| **TokenEscaper** ||| Full parity |
244+
| **Distance calculations** ||| |
245+
| **Vector normalization** ||| |
246+
| **Logging utilities** ||| Via SLF4J |
247+
| **CompressionAdvisor** ||| **GAP: SVS compression helper** |
248+
| **SVSConfig** ||| **GAP: SVS configuration model** |
249+
250+
---
251+
252+
## 12. CONNECTION MANAGEMENT
253+
254+
| Feature | Python | Java | Notes |
255+
|---------|--------|------|-------|
256+
| Basic Redis connection ||| |
257+
| Connection pooling ||| |
258+
| SSL/TLS ||| |
259+
| Sentinel support ||| |
260+
| Cluster support ||| |
261+
| URL-based configuration ||| |
262+
| Async connections || N/A | |
263+
264+
---
265+
266+
## Summary of Gaps
267+
268+
### Medium Priority (Extended Functionality)
269+
270+
1. **Cloud Vectorizer Integrations**
271+
- Missing: OpenAI, Azure OpenAI, Cohere, Mistral, Vertex AI, Bedrock, Voyage AI
272+
- Workaround: Use LangChain4J embeddings which support most of these
273+
274+
2. **CompressionAdvisor Utility**
275+
- Python: Helper to recommend SVS compression settings based on dimensions/priorities
276+
- Java: Not implemented (SVS-VAMANA core functionality IS implemented)
277+
278+
### Low Priority (Nice-to-Have)
279+
280+
3. **CLI Tools**
281+
- Python: `rvl` command-line interface
282+
- Java: Not implemented - lower priority for Java library consumers
283+
284+
4. **vectorizer_from_dict() Factory**
285+
- Python: Factory function to create vectorizers from config
286+
- Java: Builder pattern serves similar purpose
287+
288+
### Completed (Previously Gap)
289+
290+
5. **SemanticMessageHistory** - ✅ NOW IMPLEMENTED
291+
- Python: `get_relevant(prompt)` searches messages semantically using vector similarity
292+
- Java: `SemanticMessageHistory` class with `getRelevant()` method - full parity
293+
294+
---
295+
296+
## Recommendations
297+
298+
### For 0.1.0 Release (Current)
299+
300+
The current feature set is sufficient for a production-ready 0.1.0 release:
301+
- Core schema/index/query functionality ✅
302+
- Vector search (FLAT, HNSW, SVS-VAMANA) ✅
303+
- Semantic caching ✅
304+
- Semantic routing ✅
305+
- LangChain4J integration ✅
306+
- Rerankers ✅
307+
- Message History (including SemanticMessageHistory) ✅
308+
309+
### For Future Releases
310+
311+
1. **Ongoing**: Add cloud vectorizer integrations as needed (or recommend LangChain4J)
312+
2. **Consider**: CLI tools if there's user demand
313+
314+
---
315+
316+
## Test Coverage Comparison
317+
318+
| Area | Python Tests | Java Tests | Notes |
319+
|------|--------------|------------|-------|
320+
| Schema ||| Comprehensive |
321+
| Index ||| Comprehensive |
322+
| Queries ||| Comprehensive |
323+
| Filters ||| Comprehensive |
324+
| Vectorizers ||| Local models tested |
325+
| Rerankers ||| Mock/integration tests |
326+
| SemanticCache ||| Comprehensive |
327+
| LangCache ||| Comprehensive (PR #442) |
328+
| MessageHistory ||| Basic coverage |
329+
| SemanticRouter ||| Comprehensive |
330+
| LangChain4J | N/A || Comprehensive |
331+
332+
---
333+
334+
*Report generated for RedisVL4J feature parity audit.*

0 commit comments

Comments
 (0)