feat: Alternative Embedding Providers (OpenAI, OpenRouter, HuggingFace)

.env.example

@@ -1,28 +1,64 @@
-# OpenRouter API Configuration
+# Concept-RAG Environment Configuration
+
+# ==============================================================================
+# OPENROUTER API CONFIGURATION (Required for concept extraction)
+# ==============================================================================
 # Get your API key at: https://openrouter.ai/keys
 OPENROUTER_API_KEY=your_openrouter_api_key_here
 
-# Database Configuration (Optional)
-# Default: ~/.concept_rag
-# Uncomment to use a custom database location
-# CONCEPT_RAG_DB=/custom/path/to/database
+# Optional: Override default models for concept extraction
+# OPENROUTER_CONCEPT_MODEL=anthropic/claude-sonnet-4.5
+# OPENROUTER_SUMMARY_MODEL=x-ai/grok-4-fast
+# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
+
+# ==============================================================================
+# EMBEDDING PROVIDER CONFIGURATION (Optional - for production embeddings)
+# ==============================================================================
+# Choose your embedding provider: simple, openai, openrouter, or huggingface
+# Default: simple (no API key required, suitable for development/testing)
+# EMBEDDING_PROVIDER=simple
 
-# Model Configuration (Optional)
-# These are the default models used by concept-rag
-# Uncomment to override with different models
+# ------------------------------------------------------------------------------
+# OpenAI Embeddings (if using EMBEDDING_PROVIDER=openai)
+# ------------------------------------------------------------------------------
+# Get your API key from: https://platform.openai.com/api-keys
+# OPENAI_API_KEY=sk-proj-...
+# OPENAI_EMBEDDING_MODEL=text-embedding-3-small
+# OPENAI_BASE_URL=https://api.openai.com/v1
 
-# Concept extraction model (default: anthropic/claude-3.5-sonnet-20240620)
-# CONCEPT_EXTRACTOR_MODEL=anthropic/claude-3.5-sonnet-20240620
+# ------------------------------------------------------------------------------
+# OpenRouter Embeddings (if using EMBEDDING_PROVIDER=openrouter)
+# ------------------------------------------------------------------------------
+# Note: Can reuse the OPENROUTER_API_KEY from above
+# OPENROUTER_API_KEY=sk-or-v1-...
+# OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
+# OPENROUTER_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1
 
-# Summary generation model (default: x-ai/grok-2-1212)
-# SUMMARY_MODEL=x-ai/grok-2-1212
+# ------------------------------------------------------------------------------
+# HuggingFace Embeddings (if using EMBEDDING_PROVIDER=huggingface)
+# ------------------------------------------------------------------------------
+# Option 1: Use HuggingFace API (get key from: https://huggingface.co/settings/tokens)
+# HUGGINGFACE_API_KEY=hf_...
+# HUGGINGFACE_MODEL=sentence-transformers/all-MiniLM-L6-v2
 
-# Embedding model (local, no API key needed)
-# Uses Xenova/all-MiniLM-L6-v2 (384-dimensional embeddings)
+# Option 2: Use local inference (no API key required, requires @xenova/transformers)
+# HUGGINGFACE_USE_LOCAL=true
+# HUGGINGFACE_MODEL=Xenova/all-MiniLM-L6-v2
+
+# ==============================================================================
+# DATABASE CONFIGURATION (Optional)
+# ==============================================================================
+# Default: ~/.concept_rag
+# Uncomment to use a custom database location
+# DATABASE_URL=/custom/path/to/database
+# CONCEPT_RAG_DB=/custom/path/to/database
 
-# Logging Configuration (Optional)
-# LOG_LEVEL=info  # Options: debug, info, warn, error
+# ==============================================================================
+# LOGGING & PERFORMANCE (Optional)
+# ==============================================================================
+# Logging level (options: debug, info, warn, error)
+# LOG_LEVEL=info
 
-# Performance Tuning (Optional)
+# Batch processing configuration
 # BATCH_SIZE=10  # Number of documents to process in parallel during seeding
 # MAX_TOKENS=100000  # Maximum tokens per document for concept extraction

README.md

@@ -196,6 +196,72 @@ npx tsx hybrid_fast_seed.ts \
 - ⚡ Creates fast local embeddings (384-dimensional)
 - 💾 Stores in 3 LanceDB tables: catalog, chunks, concepts
 
+### 🎯 Embedding Providers (Optional)
+
+Concept-RAG supports multiple embedding providers for semantic search. By default, it uses a simple hash-based embedding service suitable for development. For production use, configure one of the following providers:
+
+#### Simple (Default - Development)
+
+Hash-based embeddings, no API required. Suitable for development and testing.
+
+```bash
+# Default - no configuration needed
+EMBEDDING_PROVIDER=simple
+```
+
+#### OpenAI
+
+Production-grade embeddings with OpenAI API.
+
+```bash
+EMBEDDING_PROVIDER=openai
+OPENAI_API_KEY=sk-...
+OPENAI_EMBEDDING_MODEL=text-embedding-3-small  # Optional, default: text-embedding-3-small
+OPENAI_BASE_URL=https://api.openai.com/v1     # Optional, for custom endpoints
+```
+
+**Cost**: ~$0.02 per 1M tokens  
+**Quality**: Excellent semantic understanding  
+**Dimensions**: 1536 (projected to 384)
+
+#### OpenRouter
+
+Access multiple embedding models via OpenRouter's unified API.
+
+```bash
+EMBEDDING_PROVIDER=openrouter
+OPENROUTER_API_KEY=sk-or-...
+OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small  # Optional
+OPENROUTER_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1  # Optional
+```
+
+**Cost**: Variable by model  
+**Quality**: High (depends on selected model)  
+**Benefits**: Multi-model access, usage tracking
+
+#### HuggingFace
+
+Local or API-based embeddings with HuggingFace models.
+
+```bash
+# API Mode (requires API key)
+EMBEDDING_PROVIDER=huggingface
+HUGGINGFACE_API_KEY=hf_...
+HUGGINGFACE_MODEL=sentence-transformers/all-MiniLM-L6-v2  # Optional
+
+# Local Mode (privacy-first, no API key needed)
+EMBEDDING_PROVIDER=huggingface
+HUGGINGFACE_USE_LOCAL=true
+HUGGINGFACE_MODEL=Xenova/all-MiniLM-L6-v2  # Optional
+```
+
+**Cost**: Free (local) or HuggingFace API pricing  
+**Quality**: Excellent for most use cases  
+**Dimensions**: 384 (native for all-MiniLM-L6-v2)  
+**Privacy**: Local mode runs entirely offline
+
+**Note**: To use a different embedding provider, add the environment variables to your `.env` file before running the seeding process. The embedding provider affects how semantic similarity is calculated during search.
+
 ### Step 5: Configure Cursor
 
 1. **Open Cursor settings** and navigate to MCP configuration